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tential effectiveness. According to the results, this course 
has proven its capability to provide the stimuli necessary for 
the transfer of basic knowledge and skills in HF T § E. Addi- 
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terminal objectives were substantiated. The conclusions pro- 
vided in this study are intended to encourage further course 
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I. INTRODUCTION 



There are many definitions available for the term "human 
factors". Simply stated, the major concern of all people in- 
volved with human factors is that of providing equipment and 
systems which people can use. Included in the various facets 
of human factors are design, test and evaluation. 

Technology has advanced rapidly during the last decade. 

The development of the people who must make use of these 
"state-of-the-art" technological wonders has not kept abreast 
of the rapid growth in this field. For this reason, there 
has been increased attention to the human factors aspects of 
the new systems. Design engineers and evaluators must be 
cognizant of this fact, so that rather than designing equip- 
ment for fellow technical wizards' use, they must take into 
consideration the attributes and capabilities of the ulti- 
mate user. [Ref. 1: pp. 1-27] Referred to in U. S. Navy 
terms, the requirement is to make it "sailor-proof". Be- 
cause the majority of sailors or non-prior service enlisted 
personnel as measured by the Armed Service Vocational Bat- 
tery (ASVAB) (Table 1) fall into the lower mental categories 
[Ref. 2: p. 281], this requirement is of major concern when 
implementing new systems or equipment for their operation. 

Failure of deployed systems are often caused by human 
induced errors. There are indications that the percentage 
of failures due to human ineptitude or poor human 
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reliability-*- may be quite high. The increasingly compli- 
cated nature of modern military systems together with 
shortages of qualified military personnel suggests that 
human-induced errors both in operation and maintenance of 
systems will increase unless more attention is given to 
this problem in the design and development phases of the 
acquisition process. [Ref. 1: p. 27] 



Table 1 

Mental Category of Non-Prior Service 
Enlisted Accessions for 1978 



Mental Population Marine Air 

Category Percentile POD Army Navy Corps Force 



I + II 


65 - 


100 


34 


26 38 


29 


45 


III 


31 - 


. 64 


61 


63 60 


68 


55 


IV 


10 - 


30 


5 


11 3 


4 


0 


Department 


of 


Defense 


[Ref. 1 : p . 


34] is 


aware of 



this problem and concurs that effective action must be taken 
to correct it. 

Existing military instructions and directives [Ref. 4] 
state that human factors engineering must be integrated into 
the standard test and evaluation plans of all military sys- 
tems. Command level documents [Ref. 5] include statements 



^Human reliability as used here is defined as the probabil 
ity that human error by either the operator or the maintainer 
will not cause a system failure or malfunction. The concept 
of system performance employed by the HFE Self-Paced Course 
is not concerned with human error that does not cause system 
failure. The evaluator concentrates on overall system per- 
formance. "The human is an integral part of a system and 
therefore his/her performance becomes a system component for 
evaluation. [Ref. 3: pp . 3-14] 
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which say that human factors aspects of equipments and sys- 
tems will be assessed. All too frequently, however, test 
directors design the evaluation plan without human factors 
tests, or with human factors testing scheduled on a "not-to- 
interfere basis" or without considering the basic "...char- 
acteristics of personnel who will operate and maintain the 
system when it is eventually deployed, i.e., muscular 
strength and coordination, body dimensions, perceptions and 
judgment, sensory capacities, native skills and capacity to 
learn new skills, optimum workload, basic requirements for 
comfort, safety, and freedom from environmental stress." 

[Ref. 1: p. 29] 

An additional problem arises when the evaluators become 
involved too late in the development and- acquisition cycle-- 
the "die have been cast"; the changes required are too ex- 
pensive in terms of time and/or dollars. Thus test directors 
are forced into the world of trade-off s --instead of receiving 
the best,. the user will get whatever the least costly compro- 
mise had to offer or whatever "hurt" the least. This often 
occurs because of the evaluator's inexperience with the design, 
test and evaluation processes and/or lack of guidance, train- 
ing, and education in the field of human factors. [Ref 1: p. 

10, 21, 31] 

Various elements of the Department of Defense and civilian 
authors have attempted to develop documents which provide hu- 
man factors engineering test and evaluation guidelines. Some 
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of these guidelines tell the planner when the evaluator 
should first become involved, others provide evaluation check- 
lists and criteria against which to measure. A variety of 
documents even tell how to perform the tests. From discus- 
sions held with various members in the test and evaluation 
field, indications are that most of these existing references 
lie in sad disuse. They add that for all the expertise in- 
volved in their development, no one has to date, provided 
the one link that would put these documents into high demand: 
basic training and education in human factors. Valuable 
documents such as Human Factors Test and Evaluation Manual 
(HFTEMAN) , Human Factors Engineering Data Guide for Evalua- 
tion (HEDGE), and Mission Operability Assessment Technique 
(MOAT), to name just a -few, require a basic knowledge of 
human factors for their effective use. Consistent with hu- 
man nature, it is easy to put off, ignore, or - down play the 
importance of those things which are not easily understood. 

Having recognized the need for basic training in human 
factors, the question then becomes one of implementation. 

The means of instruction must take into account cost effec- 
tiveness, and instruction standardized so that all intended 
users may benefit. During the 1970 's and 1980's, the use of 
the Personalized System of Instruction (PSI) has gained in 
popularity and credibility [Ref 6: pp. 1-9] because of its 
economical feasibility [Ref 6: pp. 11-16] and attention to 
different individual learning styles [Ref. 6: pp. 73-75]. 
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Critical aspects of the previously heretical concepts of PSI 
have been undergoing intensive study [Ref. 6: pp. 1-9; 'Ref. 7 
pp . 587-592] with proven positive results. The major success 
or failure of PSI is greatly dependent upon its design con- 
siderations. The subject matter expert usually knows what 
content must be included, but often times lacks the famili- 
arity and understanding necessary for PSI development to give 
all students an equal opportunity to master the content. 
Frequently, the problem lies in its method of presentation • 
and delivery. [Ref. 8: pp. 40-43] Adherence to the basic 
requirements for a PSI design and development can produce an 
extremely effective course [Ref. 9: pp. 165-170], 

Progressing with the advances of modern education and 
seeking cost effectiveness in delivering instruction, the 
HFE Self-Paced Course designers elected to develop a PSI for 
use with its small and specialized enrollment. The U. S. 

Navy has formally recognized procedures and agencies for PSI 
development, design, implementation and evaluation [Refs. 10, 
11 ]. 

It is the purpose of this thesis to investigate a human 
factors engineering self-paced course which is anticipated 
to provide the missing link- -education and training in basic 
human factors. Performance of this investigation will be 
accomplished in three phases: 
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1. Development of an evaluation plan for a self-paced 
course in human factors in accordance with CNET and CNTECHTRA 
directives . 

2. Implementation of the evaluation plan in a realistic 
environment , and 

3. Evaluation of the Human Factors Engineering Self- 
Paced Course. 

A description of a preliminary draft of the Human Factors 
Engineering Self-Paced Course is provided in Chapter II. The 
evaluation plan used for the effectiveness assessment of the 
course is discussed in Chapter III. Methodology and- results 
are addressed in Chapter IV. Chapter V presents this author's 
conclusions and recommendations. A final summary is given 
in Chapter VI. 
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II. THE HUMAN FACTORS ENGINEERING (HFE) SELF-PACED COURSE 



The Human Factors Engineering Self-Paced Course is clas- 
sified as a Self-Teaching Exportable Package (STEP) by . 
NAVEDTRA 106A. As such, this course is intended to be self- 
supporting with minimal student contact required with the 
distributing command. 

Once the course is implemented into the system, any mili- 
tary member may order the course materials from -the sponsor- 
ing command dr through standard military supply channels. 

Upon receipt of the course materials, the student may proceed 
with the lessons, working at home or during free time while 
"on the job". A recommended schedule is provided to each 
student as a guideline for meaningful and timely completion 
of the course materials. Thus the' participant proceeds through 
the lessons on a self-paced basis while continuing to perform 
duties and tasks at the parent command. 

This self-paced course has two’ formats: (1) a set of 

written manuals, and (2) a computer-based parogram planned to 
be available on the Advanced Research Project Agency (ARPA) 
network on a time-share basis. The total cost of either pro- 
gram is estimated to be considerably less in dollars (travel 
and per diem), man-hours required for training attendance, 
and man-hours lost from performing on-the-job tasks and 
duties while attending training than sending personnel to off- 
site human factors training facilities. This evaluation was 
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conducted using the written manuals because the computer 

2 

version was not accessable at the time. 

Because the written materials are so extensive, they are 
not provided as part of this thesis. However, excerpts and 
summarizations taken from the course materials covering the 
course purpose, design, description, learning objectives and 
content are provided in the remainder of this section. 

A. PURPOSE 

This self-paced course was developed to provide HFE 
practioners with an awareness of its facts, principles and 
issues. The course designers have intended that upon comple- 
tion of their course of study, each participant should "think . 
about human factors whenever they are doing their job". 
Therefore, its purpose is to bridge the gap between Depart- 
ment of . Defense (DOD) directives and instructions on HFE de- 
sign, test and evaluation and the knowledge level of the 
novice HFE evaluator. 

B. COURSE DESIGN AND STRUCTURE 

The HFE Self-Paced Course is a STEP [Ref. 10: p.140], 
consisting of a student supplement, 40 lessons, and various 



2 

The computer version has been written but time and fi- 
nancial constraints made it impossible to evaluate. Portable 
computer terminals were required for distribution at test 
sites and these were not available. The program had also not 
been entered on to the ARPA network for systemwide access. 
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military standards (MIL-STDS), handbooks (HDBKs) and docu- 
ments. The course is intended to provide the student with 
the same sort of instruction that would be found if she/he 
were the only student, of an instructor. The written manuals 
consist of eight lesson books, each containing five lessons. 
The student supplement is to be used in concert with either 
the written or computer generated format. 

The basic course has been divided into three modules/ 
sections (major subject areas), the first contains 20 lessons, 
the second 15 lessons and the third two lessons. Practical 
application is required in lessons 38 and 39 while a full 
course review is provided in the last lesson. Each lesson 
consists of a series of small blocks of information. A 
question is asked at the end of each block to check compre- 
hension and progress, providing immediate feedback to the 
student. Each question has several responses from which to 
choose and the student must select the answer believed to be 
best. The student is then either directed to a page from the 
lesson book or presented with a computer image which contains 
a comment on the selected answer stating whether it is cor- 
rect or incorrect, and why. After providing remedial infor- 
mation following an incorrect selection, the student is 
directed to choose another answer. If the answer is again 
incorrect, the cycle continues until the student selects the 
correct answer. In the written format, the order of the ma- 
terial, questions and responses is scrambled so that it is 
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not possible to skim the pages of each lesson book and there- 
by skip the progress checks. Design of the computer version 
also eliminates "skimming". There are no "End o-f Module/ 
Section or Lesson Progress Tests". 

Each lesson is developed in a story line. The central 
character is a junior officer (Lieutenant) who is unskilled 
and untrained in human factors. In the beginning, the lieu- 
tenant is characterized as having a negative attitude towards 
human factors. As the course continues, various events cause 
the officer to become a proponent of human factors. It is 
intended that the student learn from the numerous mistakes 
of the ficticious Lt. I. M. Eager and the guidance provided 
by his HFE mentor and boss, Capt. B. Smart. 

In addition to the data presented throughout the lessons, 
a student supplement has been provided. As the student pro- 
gresses through each lesson, he/she is asked to refer to the 
supplement. The supplement provides the student with graphs, 
photos and charts, some practical work (mathematics and 
drawings), and "nice- to-know" information. 

The course has been designed with the intent that each 
should take no longer than one hour to complete, with an 
average completion time of 35 - 40 minutes and should be done 
on a daily basis. Therefore, maximum total completion time 
should not exceed 40 hours, or 40 working days if one lesson 
is finished per day. It is anticipated that the computerized 
version would, in all probability, reduce the completion time 
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. 









for each lesson, as it eliminates the procedure of flipping 
through the lesson books searching for progress check re- 
sponses and new material. The- computer locates the response 
and displays it automatically for the student. 

In addition to the aforementioned course materials vari- 

% 

ous MIL-STDs, handbooks and documents are provided to the 
student and their usage is addressed throughout the course. 
Some are required for successful course completion^, while 



3 

Documents required for successful course completion: 

a. MIL-H-46885B. Human Engineering Requirements for 
Military Systems, Equipments and Facilities . Janu- 
ary 1979. 

b. MI L-STD- 1472C . Human Engineering Design Criteria 
for Military Systems, Equipments and Facilities' ! - 

c. MIL-STD-1747B. Noise Limits for Army Materiel. 
March 1975. 



d. MIL-HDBK- 759 . Human Factors Engineering Design for 
Army Materiel . March 1975 

e. Army Regulation 602-1. Personnel-Materiel Systems : 
Human Factors Engineering Program . U. S. Army 
Headquarters, Washington, D. C. June 1976. 

f. Technical Memorandum 29-76. Guide for Obtaining and 
Analyzing Human Performance Data m a Materiel De - 
velopment Project . Berson, BT E"! and Crooks , WI H. , 
U! S. Army Human Engineering Laboratory, Aberdeen 
Proving Ground, Md., September 1976. 
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others are referred to as additonal resources . While the 
basic package of' reference documents provided with the course 
materials must be returned along with the lesson books to the 
issuing agency, they may be ordered through the standard mil- 
itary supply system. A brief summary of each of these re- 
sources is provided in Appendix A. 

C. COURSE PRESENTATION 

The course materials are presented in a manner which 
allows the student to complete the course of instruction 
without the aid of an on-site instructor. The student sup- 
plement is intended to be retained by the student and used 



4 

Documents referred to but not required for course com- 
pletion: 

a. MIL- STD- 721B . Definitions of Effectiveness Terms 
for Reliability, Maintainability, Human Factors, and 
Safety ^ August 1966 . 

b. FED-STD- 595 . Colors . 26 August 1964. 

c. Holshouser, E. L. , Guide to Human Factors Engineer - 
ing General Purpose' Test Planning (GPTP"H Pacific 
Missile Test Center, Pt. Mugu, Ca. 

d. Human Factors Test and Evaluation Manual (HFTEMAN) , 
Vols . F ] II, and III. Pacific Missile Test Center, 
Pt . Mugu, Ca. 

e . Human Factors engineering Data Guide for Evaluation 
(HEDGE JT U~i STI Army Test and Evaluation Command. 

20 December 1977. 

f . Mission Operability Assessment Technique : A System 

Evaluation Methodology (MOAT) . October 1979. TP- 79- 
31. Pacific Missile Test Center, Pt. Mugu, Ca. Lt. 
W. R. Helm and M. L. Donnell, Ph.D. 
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as a "stand-alone'' resource and reference document following 
course completion. The student may contact the issuing agency 
in the event additonal help is required. 

D. COURSE LEARNING OBJECTIVES 

In accordance with the recommendations of NAVEDTRA 106A, 
a list of performance requirements was collected by course 
designers after consultation with human factors engineering 
test and evaluation personnel. These were then restated into 
task requirements (Appendix B) and forwarded to the civilian 
contractor who was assisting in the course development. The 
course learning objectives (Appendix C) were then developed 
from this list of task requirements. No module or lesson 
learning ojbectives were developed. 

E. COURSE CONTENT 

As described in the introduction of the Student Supple- 
ment, the course is divided into three major content sections 
(modules): (1) human capabilities and limitations, (2) human 

integration with the system, and (3) "Human Factors in the 
Military". 

Module I (Lessons 1-20) investigates the history and con- 
tinued need for human factors engineering. Physiological ca- 
pabilities and limitations of human beings and their 
incorporation into the proper design and controls and displays 
are addressed. Also included is a discussion of the human’s 
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interactions with the surrounding environment and the effects 
it may have upon his/her performance. 

Module II (Lessons 21-35) discusses: (i) the role of the 

human engineer in various aspects of analysis, i.e., systems, 
cost, task analysis and etc., (2) selection and training of 
personnel; and (3) experimental techniques and statistical 
concepts . 

Module III (Lessons 36-37) focuses on human factors organ- 
izations, documents, and future applications. To. provide 
students with a practical application of what has been pre- 
sented in the course, lessons 38 and 39 present a "real world" 
problem for solution. Finally, in lesson 40, a more typical 
review of the entire course is conducted. 

The outline of the Module and lesson topics is provided 
in Appendix C. 
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III. EVALUATION PLAN 



The validation process is a critical step in the devel- 
opment of lesson materials. The question to be asked is: 
"How can we be certain that the instruction works?" The 
only acceptable way to do this is to measure student per- 
formance. If the students learn the specified skill, 
task, or knowledge as a result of the instruction, then 
it is valid; if the students do not learn as specified in 
the learning objective, the instruction is not valid and 
must be revised. Validation can be compared to the test- 
ing process on a new piece of equipment. If particular 
components of the equipment malfunction repeatedly, then 
this indicates a problem that must be corrected if the 
equipment is to perform properly. By the same token, if. 
the students fail to master a particular segment of in- 
struction, a problem must be corrected. [Ref. 11] 



To define the strategy used to assess the potential effec- 
tiveness of the instructional materials for the HFE course, 
an evaluation plan was ' developed (Appendix E) , in accordance 
with the . guidelines provided in NAVEDTRA 110A. The evaluation 
plan provided in this thesis combined several of the valida- 
tion steps prescribed. These combinations were made because 
(1) no validation plan had been developed by the course spon- 
sors, (2) no previous evaluation of the course materials had 
been conducted, and (3) the course was in an advanced state 
of development. While the computer version has been written, 
it is not available on a commonly accessible system. The 
evaluation was, therefore, developed using the set of written 
manuals . 



28 



A. HOW TO CONDUCT VALIDATION 



Validation is accomplished by completing the following 
steps [Ref. 11] : 

1. Interval Review . As the rough draft materials are 
evaluated, the following questions should be asked: 

a. Is the content accurate? 

b. Is the material presented in a logical fashion? 

c. Does the lead-in information motivate the student 
to pursue the material? 

d. Do the teaching-learning activities encourage 
productive learning? 

e. Is the material written in a .manner to allow 
maximum student participation? 

f. Is there an opportunity for review and practice? 

g. How effectively does the material teach behaviors 
specified in the learning objectives? 

h. Does the test measure the behavior as specified 
in the learning objectives? 

The internal review does not determine validity of the 
course material. It does, however, identify problem areas 
and suggest alternatives. 

2. Individual Trials . Informal, individual trials on 
rough draft materials for each lesson will take place to 
identify gross deficiencies in the lesson materials "before" 
expensive final draft materials are produced. In other 
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words, these trials should help identify where more practice 
is needed, instruction is adequate, and whether instructor 
guides or student study booklets are complete. Since this is 
probably the first time students will actually take the test 
items developed for a lesson, the informal trial may also 
identify problem test items. 

The individual trials should be conducted as follows: 

a. Administer Pretests. Pretests are given to stu- 
dents to determine how much they know before being exposed 
to the instruction. The scores made .on the pretest are then 
compared with scores made on post tests to ascertain the ef- 
fect instructional materials had on student performance. 

Tests developed for validation purposes are com- 
posed of the content items that address both enabling and 
terminal objectives. For validation purposes, however, it 
may be helpful to add test items to both pre and post tests 
which would be diagnostic in nature and therefore help to 
locate problem spots in the instructional materials. 

b. Administer Instructional Materials. Observe the 
students' performances as they complete the lesson and record 
their completion time, the questions asked, and any difficul- 
ties observed. 

c. Administer Post Test Items. This will measure 
mastery of the content by the students. The results will be 
used to identify strengths and weaknesses in the course and 
to provide a basis for making revisions. 
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d. Administer Student Feedback Sheets. This is the 



student's opportunity to suggest how to make instructional 
materials more effective. As such, the feedback sheet should 
be structured so that students' responses are channeled, yet 
flexible so that open ended responses are allowed. 

e. Revise the Instructional Materials. Evaluate all 
data: test scores and instructor and student comments. Then 

make the revisions required. When all major deficiencies have 
been corrected and revalidated, the individual trials are 
concluded. 

3. Group Trials . A minimum of six students should be 
used for each group trial. As with individual trials, it is 
very important to select students that are representative of 
the target population and that have the proper prerequisites 
for the experimental group. The steps for conducting group’ . 
trials are the same as were followed during the individual 
trials with one major exception: this time, there is only 

one observer to six or more students. As students proceed 
through the instruction, they are not interrupted; they con- 
tinue through it as would a normal class. Only upon comple- 
tion of the lesson materials may students be asked such ques- 
tions as why they missed certain items, did the instructional 
materials maintain their interest, were there enough practice 
items . 



31 



4. Operational Validation . Validation of a new or revised 
course will be conducted in the operational environment using 
the normal student population. The reasons for conducting 
the operational validation are: 

a. Instructional materials must be evaluated as an 
integral part of a total system. Until now, individual and 
group trials validated portions of instructional material in 
an isolated environment. 

b. Analysis of data from this larger sample will 
provide a solid basis for final revision and refinement of 
the course. Data gathered at this point will provide feed- 
back concerning the adequacy of the learning analysis, learn- 
ing objectives, criterion tests, and instructional materials. 

If students fail to meet the standards of the terminal objec- 
tives, reassessment of each of these procedures and products 
of the course design and development processes shall be scru- 
tinized. 

c. An operational trial provides an opportunity to 
work out administrative, equipment, facility, or any other 
implementation problems which may cause trouble later. 

The steps for completing operational validation are: 

Cl) Review of Material by Functional Command . 

The functional command will review a cross-section of all 
instructional materials for content, technical accuracy, and 
format. These will include instructor guides, student study 
guides, remediation guides, media, student study booklets and 
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laboratory guides. Discrepancies noted by the functional 
command will be discussed with the curriculum designer. 

(2) Administer Pretests, Instructional Materials , 
and Post Tests . As with both individual and group trials, it 
is absolutely necessary to have students with the appropriate 
entry level; without this, validation results are contamina- 
ted. 



(3) Administer Student and Instructor Feedback 



Sheets . 

(4) Analyze Results and Display/Present Data to 
Approval Authority . After administering all the tests, in- 
struction, and feedback sheets, the results must be analyzed 
and displayed for submission to the approval authority. Pri- 
or to operational validation, a validation standard must be 
set. At this time, the validation test results are compared 
with the validation standard. If the standard is not reached, 
the curriculum designer should look carefully at the instruc- 
tional materials, equipments, and management documents to 
locate the problems and decide on corrective action to be 
taken. 



B. EVALUATION PLAN 

The format used for the Evaluation Plan was modeled after 
that used by the original test site in evaluating equipment 
and systems [Ref. 5] because of familiarity with selected 
test site supervisors. Following its design and development, 
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other military branch sites requested to be included in the 
evaluation. A- summary of the evaluation plan follows: 

1. Purpose . The purpose of the evaluation was to pro- 
vide an assessment of the potential effectiveness of the HFE 
Self-Paced Course and its potential value for use in the mil- 
itary test and evaluation community. The evaluation was plan- 
ned in cooperation with the HFE Laboratory at Pacific Missile 
Test Center, Pt. Mugu, California. 



2. Background . The HFE Self-Paced Course was developed 
to satisfy a need for increased awareness and more indepth 
understanding of human factors. The Government Accounting 
Offic§ (GAO) highlights this need and identified a defici- 
ency in the performance of various weapon systems 



...because the DOD does not pay enough attention to lo- 
gistic support, human factors and quality assurance during 
the design phase of the acquisition process. These pro- 
blems deter the systems’ effectiveness to defend our coun- 
try in case of war. GAO, therefore, makes recommendations 
to improve the management and planning of ownership con- 
siderations that have an impact on the effectiveness of 
a weapon system. [Ref. 1: pp. 1-27] 



The Human Factors Test and Evaluation Manual (HFTEMAN) 
Vols. I, II and III was distributed to various government 
agencies in October of 1976 by Pacific Missile Test Center. 
This document was developed to provide standardization in 
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procedures, testing and criteria in evaluating human factors. 
A basic knowledge of human factors -terminology , principles 
and concepts was necessary for its most effective use. This 
was generally lacking and therefore HFTEMAN was not utilized 
as it was intended. 

The HFE Self-Paced Course was developed to provide 
this basic knowledge. It has not been evaluated or used on 
a trial basis in any portion of the military prior to this 
study. 

This researcher was tasked to (1) develop an evalu- 
ation plan in accordance with NAVEDTRAs 110A and 106A, (2) im 

plement it, conducting the test in a realistic environment, 
and (3) analyze the data and provide results, recommendations 
and conclusions. Additional taskings and responsibilities 
are described in Appendix E. 

3. Prevalidation . This researcher conducted the inter- 
nal review with the assistance of content and curriculum 
development subject matter experts. 

4 . Informal/Group Trials/Operational Trial . 

a. Personnel Selection. Subjects participating in 
this evaluation were drawn from four military commands (two 
Navy and two Air Force) whose primary mission was operational 
test and evaluation of equipment and systems prior to their 
acquisition by DOD for service use. Personnel were to parti- 
cipate on a volunteer basis. Students selected from the 
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volunteers were to be representative of the target population 
which would be ultimately using the instructional materials. 

The following personnel prerequisites were established: 

Cl) Subjects were selected from various warfare 
communities, i.e., air, surface, subsurface and special war- 
fare. 

(2) No constraints regarding rate, rank, speci- 
ality code, grade, educational level or prior experience were 
imposed. 

(3) Each subject was. required to be actively 
involved in planning and performing test and evaluation. 

b. Testing Constraints. 

(1) Learning Center Instructor . No on-site 
instructors were available at any of the test sites, nor 
were they required. This researcher was designated as the 
students' contact point, temporarily fulfilling the future 
role of the sponsoring agency. A Command Course Supervisor 
[CCS) was identified at each location to assist this re- 
searcher in distributing and collecting course- materials , 
tests, feedback sheets (questionnaires), and recording stu- 
dent questions or comments and demographic data. 

(2) Personnel Selection . While HFE designers 
and evaluators are equally important, the availability of 
inexperienced equipment designers needing instruction was 
extremely limited as compared to those in equipment evaluation. 
The very dynamic and mobile nature of the course designers' 
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military target population identified the intended users of 
the course--U. S. military operational test directors involved 
in whole or in part in human factors evaluation. The original 
test plan called for the use of volunteers only, however, due 
to limited personnel availability, final analysis showed that 
some participants had been, assigned. 

(3) Course Materials . Original plans allowed 
60 days for student completion of all 40 lessons of the HFE 
Self-Paced Course. As a result of delayed procurement of 
course materials, the actual evaluation was limited to 30 
days and the first 20 lessons of the course. 

(4) Course Completion . The personnel partici- 
pating in this evaluation were also performing their normal 
jobs as test directors. Potential conflict was anticipated 
with their normal respons iblities which could require their 
absence from the command for a prolonged period of time and 
cause interruption in course progress or even termination. 

(5) - Learning Center . No area was available at 
any location which could be specifically designated for stu- 
dent use. Students studied in offices, libraries and at home. 
This was congruent with the course design and the intent of 
the study. 

(6) Testing . No tests had been developed by the 
sponsor. The tests used in this evaluation were developed by 
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the author and had received only limited pretesting prior to 
the evaluation" because of time constraints and therefore had 
not been validated. 

(7) Training . No training beyond that offered 
by the HFE Self-Paced Course was conducted during the opera- 
tional trial run of this study. 



5. Judgment Criteria . Both the pretest and post test 
were designed in accordance with the objectives developed by 
this researcher (Table 2). 



a. Pretest. The pretest was used to measure prior 
knowledge of the course content possessed by entering students. 
The pretest was composed of four different types of questions, 



each being assigned a specific 
multiple choice 
matching 

fill in the blank, 
short answer 

The scores <3 f the Course Group 
Control Group to ascertain entry 



point value for scoring: 

1 point per answer choice 

1 point each response 

2 points each 

3 and 5 points each 
re compared to those of the 
level equivalency. 



b. Post Test. The post test was administered after 
completion of 20 lessons of instruction to asses.s whether: 

(1) the students had mastered the objectives of the first 
module, and/or (2) the students taking the course knew more 
about human factors than students not taking the course. The 
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Table 2 



Learning Objectives Developed (By the Researcher) 
For Design of Pre and Post Tests 



1. Given a list of human factors terms, write a definition 
for each in your own words. 

2. Given a list of specific HFE .probems and/or areas of con- 
cerns, identify which human factors references should be 
consulted to provide solutions and guidelines. 

3. Given at least one control/display example, use the ap- 
propriate human factors reference to determine which 
features need to be redesigned. 

4. Given a list of potential sources of technical informa- 
tion on human factors, identify the general purpose of 
each. 

5. Explain in your own words the importance of evaluating 
the human factors aspects of systems and equipments. 

6. Identify the human factors principles which must be ap- 
plied during the evaluation of a workspace and control/ 
display panel. 

7. Given the four phases of the Weapons Acquisition Cycle, 
describe the HFE specialist role, the major HFE concerns 
and objectives of testing for each phase. 

8. Given the mission statement and the three functions which 
must be performed for a particular system, determine the 
human performance requirements. 

9. Given a list of 10 factors and forces which affect human 
performance* identify what aspect of human performance 
each affects. 

10. Given a list of 10 factors and forces which affect human 
performance, identify what unit of measure must be used 
to assess the amount of protection required to diminish 
their adverse effects upon the human, operator. 

11. Explain in your own words, the importance of performing 
a task analysis. 

12. Given a choice of 3 possible task analyses, correctly 
identify which satisfies the definition of "task analysis". 
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post test was composed of the same type of questions and the 
associated point values as the pretest; however, the propor- 
tionment of the four different types of questions was dif- 
ferent so as to reduce "chance" selection of correct answers. 
The post test was designed to test the same learning objec- 
tives as the pretest. The post test scores of the Course 
Group were compared to those of the Control Group to deter- 
mine whether or not learning had occurred as indicated by 
the Course Group’s scoring significantly higher than the 
Control Group. 

c. Test Items. Each test item was "referenced" to 
a specific terminal objective. Since each objective was a 
statement of a "criterion" for student performance, these 
tests for objectives were identified as "criterion-refer- 
enced" tests. The test items were developed in accordance 
with the guidelines provided in NAVEDTRA 110A. The questions 
and answers were extracted from the Human Factors Engineering 
Self-Paced Course, lesson 20 which contained review material 
covering lessons 1-10. Student responses which expressed 
the same answers as the course materials were given credit. 
Partial credit was awarded where possible. Test key answers 
were compared to the answers provided by selected HFE experts 
to ensure the HFE course answers were, in fact, correct. 

Additonally, to help identify weaknesses, test 
items were added to both pre and post tests which were 
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diagnostic in nature and therefore would help locate problem 
areas in the instructional materials. Results were recorded 
for each test item. 

d. Hypothesis for Comparison. If there was no 
statistically significant difference between the post test 
scores of the group completing the first module and the scores 
of the Control Group which did not take the course, it would 
indicate that the course materials were not effective and 
would be of no value for use in the military. 

Further analysis would then be conducted on both 
the pretest and post test questions. 

e. Statistical Analysis. The statistical analysis 
was performed on the pre and post tests and was intended to 
establish (1) the entry level equivalency of both the Course 
Group and the Control Group based on the pretest scores and 
(2) to establish a difference in the post test scores between 
the two groups. The results of the post test scores would 
therefore provide an overall statement of the potential 
effectiveness of the HFE Self-Paced Course. 

f. Raw data for the pre-post test analysis for the 
operational tryout was provided to this author in conjunction 
with students’ tests and is presented in summary form (Appen- 
dix G) . 
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6. Standards 



Criterion to establish equivalency between both the 
Course Group and the Control Group on the pretest scores had 
been set by this author. There must be no statistically sig- 
nificant difference at the .025 level (two-tail t test) to 
determine that the groups are equivalent in their pre-entry 
level . 

Criterion to establish that learning has taken place 
with the Course Group had been set by this author. There 
must be a statistically significant difference between the 
Course Group and the Control Group post test scores, at the 
.025 confidence level (two-tail t test) to determine that 
the groups are not equivalent in their level of learning. 

To validate that the course materials were able to success- 
fully provide the stimuli for knowledge and skills transfer 
to the students 'taking the course, their post test scores 
must be significantly greater than the Control Group. 

Criterion guidelines were established by CNTECHTRA 
for acceptability of course objectives, design, presentation, 
availability, maintainability and supportabil ity in NAVEDTRAs 
110A and 106A and are summarized by this researcher in Table 
3 and described in greater depth in Appendix E. 

In the event the standards are not met following the 
operational tryout, instructional materials would be analyzed 
as indicated above. 
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Evaluation Areas, Objectives, Criteria and Data Sources 
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Table 3 - Continued 
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Table 3 - Continued 
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7. Remediation 



Students involved in the operational tryout receive 
remediation from the responses to the Progress Checks follow- 
ing each block of instruction within the lesson topics. 

8 . Instructor Training 

The HFE Self-Paced Course was intended for use by 
individuals without the aid of an on-site instructor. The 
intent of the agency sponsoring the course is to provide a 
point of contact for students to answer any questions on con- 
tent, procedures, etc., once the course is approved. There 
was, however, no instructor guide available during this study. 
If none is to be developed, the intended point of contact 
must then be a subject matter expert. The student and the 
expert would then talk through any problems using the set of 
student course materials. 

9 . Student and Instructor Feedback Sheets (Questionnaires ) 

The following student, Command Course Supervisor and 

Curriculum Development Expert, feedback questionnaires were 
completed and returned to this author for review and analysis. 

a. Annex (A) of Appendix E was completed by each 
participant in the evaluation during orientation. 

b. Annex CC) of Appendix E was completed by each 
participant prior to the students beginning the course. 

c. Annex (D) of Appendix E was completed by each 
participant after the Course Group had completed the first 
20 lessons. 
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d. Annex (B) of Appendix E was maintained by each 
Command Course Supervisor (CCS) throughout the evaluation 
period. 

e. Annex (G) of Appendix E was completed by experts 
in the field of curriculum development. 

f. Annex (F) of Appendix E was completed by each 
Course Group participant upon completion of the post test. 

g. Annex (E) of Appendix E was completed by each CCS 
upon completion of the evaluation period. 

10 . Display Format 

Test analysis results are displayed as Appendixes F 
and G and discussed in Chapter IV of this thesis. 

Analysis of the questionnaires are displayed in 
Appendix H and discussed in Chapter IV of this thesis. 
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IV. METHODOLOGY AND RESULTS 



For this study 52 participants were chosen from four 
test sites, each of whom met the prerequisites for course 
entry as described below. These 52 participants were divided 
into two groups; one which would complete the course (Course 
Group) and the other (Control Group) which would not go 
through the course, but would be administered the pre and post 
tests. Since the pre. and post tests were to be structured 
differently, the post test scores could not be compared to 
the pretest scores to measure the effectiveness of the course 
in providing the stimuli for transferring knowledge and skills 
in human factors. It was, therefore, necessary to compare 
the post test results of the Course Group to those of a 
Control Group who had not completed the course of study. 

For that comparison to be valid, it was necessary to estab- 
lish an equivalency of the entry levels of the two groups 
prior to' commencing the course of .instruction. This measure 
of equivalency included data gathered from demographic infor- 
mation sheets completed by each participant and pretests on 
the subject material. 

Of the original 26 students beginning the course, 10 were 
unable to complete the 20 lessons due to conflicts of tasks 
imposed by their job responsibilities. Therefore, only 16 
post tests from the original 26 students in the Course Group 



48 



were used for the comparison. All 26 post tests from the Con- 
trol Group were used. A breakdown of experiment participants 
and data sources is provided in Table 4. 

The comprehensive examinations used for the pretest and 
post test were prepared by this researcher (Appendix E, An- 
nexes C and D) and structured as described in section III of 
this thesis. While the post test had the same type of ques- 
tions as the pretest, there were a larger number of short 
answer and essay type questions on the post test. This tend- 
ed t-o eliminate a large portion of guessing and required 
that the students have a more comprehensive knowledge of the 
course material. [Ref. 11: pp . 3-30] Although no claim is 
made as to the validity of these examinations, they do 

m 

represent a measure of student knowledge and achievement in 
relation to stated learning objectives of the course. 

A. DEMOGRAPHIC DATA, ANALYSIS AND RESULTS 

No prerequisites were required of students prior to 
taking "the HFE Self-Paced course. A prerequisite established 
by this researcher for this study required that each student 
be actively involved in test and evaluation for motivational 
purposes . 

A Student Information (SIF) sheet was distributed to and 
completed by each participant. Specific information included 
pay grade, educational level, time stationed at the test sites, 
job description and previous courses completed in human factors 
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Breakdown of Experiment Participants and Data Sources 
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Learning 
Obj ec- 
tives 
Experts 



training. This demographic data for each participant is pro- 
vided in Appendix F. A comparative breakdown (Table 5) be- 
tween both groups was performed to specifically identify 
demographic characteristics which could bias the results of 
this study. 

The distribution of enlisted personnel and officers 
between both groups was disproportionate with the Course 
Group having 50 per cent enlisted and 38.5 per cent officers 
compared to the Control Group's 23.1 per cent enlisted and 
65.4 per cent officers. The educational level of the two 
groups was also unbalanced with 80.8 per cent of the Control 
Group having college and/or postgraduate degrees compared 
to the 57.7 per cent of the Course Group. Specifically sig- 
nificant are the percentages of those having only high 
school diplomas; 3,8 per cent of the Control Group, compared 
to 34.6 per cent of the Course Group. Additionally, the 
Control Group had an average of 35.9 months experience in 
test and evaluation while the Course Group averaged only 17.2 
months. There was also a disparity in the number having 
participated in the test director's course offered by two Of 
the test sites; 76,9 per cent for the Control Group and 53.8 
per cent for the Course Group. All participants in this 
experiment were actively involved in test and evaluation. 

These results caused some concern that test scores would 
be biased since the Control Group as a whole seemed more 
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Comparative Summary Breakdown of Demographic Data 
Between the Control and Course Groups 
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spons 



experienced in performing test and evaluation, educated to 
a higher level, and- more experienced in middle management 
responsibilities for test and evaluation. The actual re- 
sults of the pre and post tests, however, did not substanti- 
ate this concern. 

Additionally, the demographic percentages between the 
original 26 and the final 16 members of the Course Group who 
completed all 20 lessons vary insignificantly. 

B. PRE-POST TEST ANALYSIS AND RESULTS 

When all test participants had completed the pretest, the 
combined scores of all participants (Appendix F) in each of 
the two groups were compared. The mean score for the Control 
Group was 70.19 with a standard deviation of 14,85. The 
mean score for the Course Group was 64.73 with a standard 
deviation of 18.73. The following pooled varian-ce formula 
for the t-test was used to test the significance of the dif- 
ference in group mean: 



t = 





Popham and Sirotnik state that this formula will result in a 
t-value which may be interpreted with more degrees of freedom 



than other t-formulas. [Ref. 12: p. 139] Therefore, since a 
smaller t-value is needed to reject a given null hypothesis 
when a greater number of degrees of freedom are used, this 
formula produces a t-test that is more likely to be signifi- 
cant and, as a result, it is a more powerful test. 

Hypothesis one assumes that the students composing the 
two groups do have an equivalent knowledge level prior to the 
entry of the experiment and, therefore, there would be no 
significant differences on the pretest totals of the two 
groups. The results of the analysis of scores are presented 
in Table 6. 

The computed t-value was compared with a distribution 
of t-tables from the Students t- table. The values found 
indicated that for 50 degrees of freedom, the t-value of 1.74 
at the .05 level of significance- should be used. [Ref. 13] 

The computed value of t for null hypothesis one has led 
to its rejection and acceptance of the alternate. Therefore, 
there was no significant difference between the pretest scores 
of the Control Group when compared with those of the Course 
Group. • Therefore, comparing the performance of the two 
groups on the post test to detect any improvement by the Course 
Group is acceptable. 

Hypothesis two states that there would be no equivalency 
between the post test scores of the Control Group and those 
of the Course Group, with the scores of the Course Group being 
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Summary of Data and Analysis for Pretest Scores for the 
Gontrol Group and the Course Group 
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greater than those of the Control Group by at least one 
• standard deviation. This hypothesis assumes equivalency in 
the scores on the pretest by both groups. 

When all participants had completed the post tests, the 
combined scores for both goups were compared. The mean post 
test score for the Control Group members was 49.70 with a 
standard deviation of 16.48 . The mean post test score for 
the Course Group students was 93.39 with a standard devia- 
tion of 25.14. This analysis (Table 7) resulted in a sig- 
nificantly high t-value. The difference between the Control 
Group and Course Group was demonstrated and therefore the 
null hypothesis two was rejected. . 

The post test results derived from cummulative scores of 
all participants (Appendix G) are provided for further study 
and test development. A total of 176 points was possible on 
the post test which covered content in the areas of (1) task 
analysis, (2) selection of personnel for testing, (3) work- 
space design and arrangement, (4) control panel design, 

(5) individual information processing capabilities, (6) envi- 
ronmental factors which may affect human performance - units 
of measure, (7) environmental considerations inclusive in 
design, (8) objectives of taking a human factors course, and 
(9) factors contributing to human error. 

A summary of these results for the Course Group (Table 8) 
shows that all content area scores were at least 50 per cent 
of the total possible points with the exception of "Control 
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Summary of Data and Analysis for Post Test Scores for the 
Control Group and the Course Group 
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Summary of Content Area Cummulative Scores for Post Tests 
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COURSE GROUP (n=16) II CONTROL GROUP (n=26) 
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Panel Design", "Individual Processing Capabilities", and "En- 
vironmental Factors Affecting Human Performance" which 
were rated as 46.38 and 48 per cents respectively. 

A summary of these results for the Control Group (Table 8) 
shows that only one content area (Factors Which Cause Human 
Error) scored at least 50 per cent. All others scored less 
than 50 per cent with a range of 3-48 per cent. 

Additionally, the test questions were redistributed under 
each of the enabling objectives (Table 2) which they support- 
ed : 

1. Given a list of human factors terms, write a defini- 
tion for each in your own words. 

2. Given a list of specific HFE problems and/or areas 
of concerns, identify which human factors references should 
be consulted to provide solutions and guidelines. 

3. Given at least one control/display example, use the 
appropriate human factors reference to determine xriiich features 
need to be redesigned. 

5. Explain in your own words the importance of evaluat- 
ing the human factors aspects of systems and equipments. 

6. Identify the human factors principles which must be 
applied during the evaluation of a workspace and control/ 
display panel. 

9. Given a list of 10 factors and forces which affect 
human performance, identify what aspect of human performance 
each affects. 
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Summary of Test Objective Cummulative Scores for Post Test 





♦ 

T3 




co 


C> 


rH 


oo 




d 

1— H 


< 


• 


• 

\D 


• 

rH 


• 

CM 








rH 














+ 


+ 


+ 


+ 








LO 


o 


LO 


vO 




o\° 


<1 


CM 

• 


LO 

• 


o 

• 


rH 








+ 


+ 


+ 


+ 




♦ 


• 




CM 


LO 


bO 






bO 


• 


• 


• 


• 






> 


vO 




CO 


OO 




*-H < 


rH 














OO 


rH 


LO 


o 


/ V 




ONP 


CM 


O 






vO 






• 


• 


• 


• 


CM ' 














II 




• 










c 


• 


(f) 


p- 




VO 


CM 


V — ' 


in 


X 


bO 


vO 








+-> 


o 


LO 


bO 


LO 




O. 


Cl, 


P H 


rH 








CD 

o 


• 


a) 


OO 


o 


o 


O 


pc 


4-> 


HD 


♦ 


# 


• 


• 


cc 


O 


aS 


LO 




vO 


vO 




H 


s 


<M 






rH 


-4 










CM 


<M 


o 




• 










PC 


• 


V) 












</) 


(f) 


COOOOOOONCOOOOOtO^t 


H- ^ N 


\D O 


(MOO 


2 


4-> 


O 


Nr v -[ v 't0'0oot s -r v 't v -t^'0 


CD O O LO 


LO co 


OO tO bO 


O 


CL, 


PU 


H N H rH bO 


rH rH rH 


rH bO 


rH rH rH 


CJ 
















# 


0 


LOLOLOOoOLOOOOOO 


O O O O 


o o 


LO LO O 




PL 


H3 




• • • • 


• • 


• • • 






03 


coOM-LOrHObOpJCMOP- 


O CM CM O 


OO oo 


O O LO 






CE! 


bO bO to N H N P- 00 




co ^r 


CM OO O 








rH 




rH 


rH 




♦ 


♦ 


bO 


rH 


vO 


rH 




T3 


bO 


• 


• 


• 


• 




C 


> 


rH 




o 


rH 




i — i 


<; 


bO 




rH 


rH 








bO 


rH 


o 


LO 


/ V 




o\o 


LO 


LO 


LO 


VXD 


vO 






# 


• 


• 


• 


rH 














II 




• 










c 


• 


in 




*3- 


vO 


(M 


V j 


V) 


tn 




CM 


bO 


p^ 




■P 


o 


cn 


CM 


bO 


CM 


CL, 


o. 


a. 










ZD 














O 


• 


0 


LO 


O 


LO 


00 


PC 


4-> 


TD 


• 


• 


• 


• 


u 


O 


oS 


rH 


^r 


CO 


p- 




H 




o 


rH 


vO 


p- 


w 






LO 


rH 


rH 


rH 


GO 




• 










oc 


• 


</) 










CD 


V) 


t/> 


oooocooonooooooo^ 


^ ^ ^ CM 


VXD O 


CM O O 


o 


p 


o 




vO vO \0 tO 


co m- 


rH OO OO 


CJ 


a. 


a. 


HH CM 




CM 


rH 




• 


0 


lolooop-oocdoooo 


o o o o 


O LO 


OO O O 




a 


TS 




• • • • 


• • 


♦ • • 




p 


_a5 


OOHLOtObON^rOHbOQ 


H- O O H- 


OO rH 


CM VXD CO 




u 




bObOCMbOvXDLOCMCMrHLO^ 


tO bO bO H 


^ CM 


LO vO LO 








H 




rH 








) li 










t/i 


*r—% 












+j ^ 


C> 




rH 


P- 




G. O 


LO 


rH 


CM 


rH 


















U) 


3t= 












P O' 


tOlOtOWOMOtOtOin^t 


^ ^ ^ CM 


\D LO 


P- LO LO 




CL, 




rH rH 




»—H 








=te 


rHCMbObOLOvOrHCMbOi-H^- 


N H H H 


p- oo 


VXD rH CM 






O' 


< CQ Q W O 


CQ U P Ph 


Q 


Q CQ Z 






# 

•r-> 


rH 


CM 


bO 


LO 






o 











61 



Table 9 - Continued 
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10. Given a list of 10 factors and forces which affect 
human performance, identify what unit of measure must be used 
to assess the amount of protection required to diminish their 
adverse effects upon the human operator. 

All the scores for the Course Group (Table 9) rated at 
lease 50 per cent with the exception of Objective 6 which 
rated 45 per cent and Objective 9 which rated 49 per cent. 

The Control Group scores (Table 9) failed to reach 50 
per cent with the exception of Objective 6 which rated 51 
per cent. 

Since the test questions had received only minimal pre- 
testing, the results warrant further test item development 
and evaluation. Although no specific conclusions will be 
reported based on this preliminary analysis, it can be 
noted that the Course Group obtained higher scores than did 
the Control Group in all but one of the content areas and in 
all of the enabling objectives. (See right columns of Tables 
8 and 9) . 

C. QUESTIONNAIRE ANALYSIS 

Three separate questionnaires were designed and adminis- 
tered in accordance with NAVEDTRAs 106A, 110A and the Ques- 
tionnaire Construction Manual developed for the Ft. Hood Field 
Unit by the U. S. Army Research Institute for the Behavioral 
Social Science. Summary information on the design and use 
of each questionnaire is provided in Table 10. Every effort 
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Summary of Questionnaire Design and Use 
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was made to include questions on the rating forms that the 
specific users were best qualified to answer. 

1 . Student Attitude Survey (Questionnaires) 

Students who had been able to complete at least five 
lessons were required to complete the Student Questionnaire 
on various aspects of the self-paced instruction received. 
There were 24 students in this classification. Students were 
directed to answer only those questions that were applicable 
to the material they completed. Therefore, the reader will 
note that the total number of responses for each question may 
tend to vary. 

Rating scales employed in the student questionnaires 
were based on a balanced five-point Likert Scale. The num- 
ber of students selecting each descriptor was tabulated. 
Ordinal values of 1 to 5 were given to student ratings, with 
S being the most positive and 1 being most negative. These 
ratings were then divided into three groups and interpreted 
as "positive (5-4), "neutral" (3), and "negative" (2-1). 

The frequency of students responding to the two most positive 
responses was combined as was the frequency of responses for 
the two most negative. These figures were then translated 
into a percentage of the total number of students responding. 
The mean and standard deviation of the students' responses 
for each question were also calculated using the formulas 
listed below: 
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n 




S.D. = 




The mean represented the degree of positiveness of the Course 
Group's responses to each question. These responses were 
used to measure their attitudes and assist in the assessment 
of the various aspects of the course and administrative pro- 
cedures such as : 

a. Course Content and Objectives 

b. Course Design 

c. Course Presentation 

d. Student Supplement 

e. Resource Documents 

f. Human Factors Attitudes 

g. Miscellaneous 

2 . Command Course Supervisor (CCS) Questionnaire and Log 
Once CCS was selected from each of the four partici- 
pating commands. The CCSs maintained a log throughout the 

+*» 

evaluation period and upon its completion filled out a 
questionnaire . 
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Rating scales employed in these questionnaires were 
based on a numerical cell interval frequency scale. The fre 
quency of responses was grouped into three cells: 



0 - 


40 % 


41 - 


60 % 


61 - 


100 % 



The CCSs estimated the frequency of student questions and 
comments that dealt with specific areas for each of several 
pertinent sections. The purpose of the data collected was 
to verify and clarify student questions, comments and sub- 
stantiate the use of the course materials without the aid of 
an on-site instructor. CCSs were also directed to assess 
student attitudes and behaviors during the evaluation 
based upon their observations, student interactions and log 
entries. These responses were used to assist in the assess- 
ment of the following areas: 

a. Lesson Book Materials 

b. Student Supplement 

c. Resource Documents 

d. Motivation and Attitudes 

e. Student Information 

f. Additional Comments 



3 . Curriculum Development Exy>ert Questionnaires 

Three proven specialists in the field of curriculum 
evaluation who regularly apply the standards and criteria 
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provided by NAVEDTRAs 106A and 110A were asked to assess the 
extent that each of the terminal objectives satisfied the 
six criteria identified in Appendix E. The average number 
of years of experience for these experts was 5.5 years. All 
were civilians in government service whose full time job was 
to evaluate various aspects of curriculum prior to its vali- 
dation and approval for use in the instructional systems of 
Navy schools. 

Rating scales employed in this questionnaire were 
based on a balanced four-point Likert Scale. No neutral 
response was provided. ' This scale measured the degree to 
which each terminal objective met each of the six criteria 
listed in Appendix E. Ordinal values ranging from 1 through 
4, were applied to the verbal descriptors as follows: 

4 3 2 1 

/ / / / 

Completely Mostly To Some Not At 

Degree All 

D. INTERNAL REVIEW AND QUESTIONNAIRE RESULTS 

The results for the Internal Review summarized in 
Chapter IV and specifically discussed in Appendix E are based 
on questionnaire responses from students and curriculum de- 
velopment expert.^ as well as this researcher’s personal eval- 
uation. 

Hypothesis three assumes that the Course Terminal Objec- 
tives, Design, Content, and Presentation would not fail to 
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meet the standards and criteria in accordance with NAVEDTRAs 
110A and 106A. 

Hypothesis four assumes that the course materials would 
not adversely affect student attitude regarding the need for 
human factors engineering test evaluation. 

As is evidenced in the results which follow, the alter- 
nate hypothesis three for Course Design is rejected with the 
failure of the Learning Objectives to follow a logical pro- 
gressive sequence which is supportive of the order of the 
content and lesson topics in Module 1. The layout and format 
are deficient in the following areas: (1) information prior 

to the regular progress checks at the end of the "information 
blocks", (2) remediation for selection of "wrong answers", 

(3) professionalism in the quality of the story line, (4) in- * 
formation in the student supplement to allow its independent 
usage as an HFE test and evaluation planning guide, and 
(5) ease and usability of the course lesson books caused by the 
"page flipping" aspect of the course procedures. 

The null hypothesis three for Course Content, Presentation., 
and Design (sequencing of knowledge and skills factors con- 
tained in Module one) is rejected in as much as these areas 
satisfied the criteria stated in Appendix E. However, while 
the lesson topics in Modules 2 and 3 are not officially part 
of this review, it must be noted that the content failed to 
provide prerequisite knowledge and skills required for under- 
standing and demonstrating follow-on concepts and skills. 
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The alternate hypothesis four was rejected in a much as 
92 per cent of the students upon course completion, responded 
that HFE test and evaluation was "extremely important" (Ap- 
pendix H) . The students' attitudes toward human factors 
testing were not adversely affected. 

1 . Course Learning Objectives (LOs) 
a. Job Task Analysis (JTA) 

No approved JTA was used to develop or evaluate 
the LOs, as none was available. The sponsoring military 
commands developed a list of task requirements (Appendix B) 
obtained from interaction with human factors engineers. This 
list was provided to the course designers to guide the 
development of the course. This procedure is acceptable 
when no JTA exists. 

Students assess the importance of the LOs by 
identifying those which they considered necessary in the 
planning and designing of HFE test and evaluation. The 
results are summarized in Table 11. With the exception of 
LOs 3, 4, 8, 10, 14, and 17 all the remaining LOs were judged 
by at least 40 per cent of the students to be important in 
planning and designing for HFE test and evaluation. 

Students also assess the importance of the LOs 
by identifying those which they considered necessary in the 
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Student Attitudes Regarding the Importance of the 
Learning Objectives in Human Factors 
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performance of their job, which for most was general test 
and evaluation. [Ref. 3: pp. 36-38, 79-80] The results 

are' summarized in Table 12. With the exception of LOs 8, 10, 
11, 12, and 17 at least 40 per cent of the students judged 
the remaining LOs as important, 
b. LO Criteria 

The Curriculum Development Experts (CDEs) were 
given a questionnaire which provided a list of course LOs and 
asked to rate the extent to which each LO met the six previ- 
ously specified criteria (Table 13) . According to the results 
of their evaluation, no objective met the criteria "Mostly" 
or "Completely". It would appear that the LOs did not 
(1) state the objective in terms of the learner, (2) state 
standards, and (3) state the conditions under which learning 
is to occur. Most of the LOs were not behaviorally specific. 
Raw data is provided in Appendix I. 

The alternate hypothesis three is therefore re- 
jected with the failure of the course terminal objectives 
to meet the criteria identified in NAVEDTRA 106A and the 
absense of module (unit/section) and lesson enabling objec- 
tives . 

2 . Course Design 

a. Sequence and Structure 

Of primary importance in support of the "building 
block" concept of course development is the logical progres- 
sion of knowledge and skill factors addressed by the terminal 
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Student Attitudes Regarding the Importance of the 
Learning Objectives in General Test 
and- Evaluation 
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objectives and the course content. This also assumes that 
the terminal objectives- will be' presented in the same pro- 
gressive order as the lesson topics. 

To conduct this evaluation, the intent of purpose 
of each learning objective was identified and broken down by 
this researcher into four areas: ( 1 ) knowledge of concepts, 

(2) knowledge of skills, (3) application of knowledge and 
(4) application of skills. The HFE course materials were 
then reviewed, identifying lessons which satisfied the in- 
tent or purpose of each objective (Table 14). NAVEDTRAs 110A 
and 110A require that a student be tasked to perform a skill 
only after the prerequisite knowledge factors have been pre- 
sented. This breakdown, therefore, enabled this researcher 
to evaluate and compare the logical progression of the termi- 
nal objectives and the lesson topics. Additionally, student 
attitude survey responses were assessed. 

In this researcher’s opinion, all 20 lesson 
topics within the first module appear to be supportive of 
the "building block"' concept for development of knowledge 
and skills. Fifty eight and- 66 per cent of student responses 
(Table 15) to questions B1 and B2 respectively are supportive 
of this opinion. In completing lessons beyond the first 
module, however, some discontinuity appears, i.e., statisti- 
cal concepts Clessons 33 and 34) such as probability of 
success, confidence levels and probability of failure-free 
system operation are addressed in lessons 22 and 26 
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respectively. Additionally, technical references (.lessons 
36 and 37) such as HFTEMAN, HEDGE and MOAT are introduced 
in lessons 23 and 24. Students are told that more detailed 
information regarding these documents will be presented 
later. This researcher needed more information about these 
documents to clarify points being made in lessons 23 and 24 
and was required to search for the information in lessons 36 
and 37. It, therefore, becomes apparent to this researcher 
that modules two and three need to be reviewed to ensure 
that a logical progression of the knowledge and skills 
factors is provided. 

In this researcher's opinion, the terminal ob- 
jectives do not follow a logical progression as exhibited by 
Table 14. Specifically, LO 3 which required students' 
knowledge of HFE technical documents as presented in lessons 
36 and 37, is sequenced before LO 4 which addresses HFE goals 
as presented in lesson 3. Therefore, LO 3 is out of sequence 
or the material in lessons 36 and 37 should be presented 
earlier in the sequence (prior to lessons 23 and 24). Table 
14 presents other examples of incongruencies of lesson 
materials. Additionally, at least 50 per cent of the students 
responding to question A1 (Table 16) felt that objectives 1, 

2, 4, and 7 were addressed within the first 20 lessons. 

Since the first 20 lessons are judged to be logi- 
cally progressive and the LOs are not, they therefore do not 
coincide in a supportive relationship. As additionally 



82 



Student Attitudes Regarding the Learning Objectives 
Taught in the First 20 Lessons 
of the HFE Course 
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required by NAVEDTRA 106A, each objective must be taught with- 
in the course of instruction. LO 11 required the student to 
'•Calculate human performance reliability” but the course 
does not teach the student how to do so and it was not one 
of the pre-course requirements. 

b. Figures and Tables 

All figures and tables used in the Student Sup- 
plement were rated by at least 65 per cent of the student 
Course Group as '’positive” (Table 17). Written comments 
were received from participants which stated that while 
figures and tables were good, without the lesson books to 
provide additonal explanation, many of them could not be 
fully understood. 

c. Symbols and Legends 

All symbology and legends used in the Student 
Supplement were rated by at least 63 per cent of the Course 
Group as "effective” (Table 18) . No additonal comments were 
received from students. 

d. Layout and Format 

Results of student attitudes regarding the 
effectiveness of the layout (Student Supplement, eight lesson 
books, and resource documents) are displayed in Table 19. 
Comments indicated that it did not hinder their learning 
experience and many indicated they preferred the way the 
lesson topics were divided into separate lesson books. They 
remarked that this added to the portability of the course 



84 



Student Attitudes Regarding Effectiveness 
of Figures and Tables 
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Student Attitudes Regarding Effectiveness of 
Symbology and Legends 
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Student Attitudes Regarding Effectiveness of 
Layout and Format 
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materials. Additonally, the students felt that much of the 
material in the Student Supplement had to be clarified by 
information in the lesson books. It would therefore,' be 
very difficult to use the Student Supplement as a stand-alone 
reference and planning guide for HFE test and evaluation. 

They considered its value as negligable once the student 
lesson books were returned to the sponsoring agency upon 
completion of the course. 

Regarding the format, no less than 50 per cent 
responded positively to (1) Course Procedures, (2) Informa- 
tion Blocks, and (3) Quality of Questions. It must be noted 
that both course procedures and quality of story line re- 
ceived a negative rating by 42 per cent of the students. 
Written and verbal comments from participants indicated a 
strong dissatisfaction with "page flipping" aspect of the 
course procedures to search for answers as well as an equally 
strong dislike for the story line following the antics of the 
ficticious Lt. I. M. Eager. 

Addtional- amplifying information regarding layout 
and format were extracted from written comments on the ques- 
tionnaires and verbal comments recorded by the CCSs. These 
have been paraphrased or quoted directly when found to be 
representative of the majority of the course participants. 

(1) As a group, course participants felt that 
the information frames sometimes did not provide all the in- 
formation necessary to answer correctly the questions at the 
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end of the information frame. In fact, comments reported that 
sometimes the information was not given until the following 
frame. 

(2) The reinforcement provided in the "wrong 
answer frames" was judged to be borderline. Students also 
commented that the wrong answer replies were demeaning, i.e., 
"Surely, you didn't answer that way". "A simple 'correct' 

or 'incorrect' with some amplifying information will suffice." 
was a comment which was representative of a majority of the 
students . 

(3) The quality of the questions asked at the 
end of the "information frames" was considered borderline. 
Comments from students indicated that sometimes the questions 
could be answered by elimination of "ridiculous answer choices" 
while others had to be guessed.- Some participants felt 

that the questions were unclear or worded poorly so as to 
cause confusion. 

(4) Several students responded that problems 
such as those already mentioned as well as errors in the 
course materials caused endless loops resulting in irritation 
and loss of motivation for continuing with the course. As 
one student stated "I would have quit by lesson five, if I 
hadn't been assigned to complete the course." 

(5) While student questionnaires indicated the 
format was neutral, fifteen of the 24 students polled chose 
to write comments concerning their dissatisfaction with the 
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"page flipping" format. They reported losing their places 
and having to begin again from frame one of the chapter being 
studied. This format may have prevented looking ahead, but 
it most certainly "prevented review of any materials". Judg- 
ing from the forcefullness and emotion emanating from actual 
comments, the following quote sums it up, "In all the service 
schools and courses which I have attended and taken in the 
past years, programmed texts were an integral part of the 
curriculum. I have never seen a programmed text structured 
such as this one. In all honesty, after the first 25 pages, 
my interest level dropped to zero and remained there. Thumb- 
ing through numerous pages to find correct answers, or incor- 
rect answers and then constantly repeating the procedure... 
created a frustrating situation that stifled any learning. 

This page shuffling drill detracted from any tangible bene- 
fits that might have been derived from the material presented." 

(6) The inclusion of- a story line following the 
adventures of the ficticious Lt. I. M. Eager and his buddy- 
boss, Capt. B. Smart was not accepted well by course parti- 
cipants. Of all the questions answered, the effectiveness 
of the story line was rated the lowest. While the numerical 
value of the mean places it at the "neutral" point, it does 
.so by only .04 of a point away from "negative". 

(7) Fourteen of the students reported that the 
story sequence was sometimes disjointed and very much dis- 
tracting from learning the essence of the lessons. Overall, 



90 



they indicated a desire to "drop the junk and put more facts 
"and real life examples in its place." "The stories made 
the text too 'cute*, don’t try to entertain me in a course, 
try to teach me. The way the material was presented (using 
Eager) just irritated me and the effectiveness of the course 
was diminished." 

3. Course Content 

In addition to the overall effectiveness assessment 
provided from the diagnostic test results, an important indi 
cator of the' perceived value of the course content was the 
number of students desiring to keep copies of the Student 
Supplement and the lesson books (Table 20). Forty-two per 
cent of the course participants desired to keep a copy of 
the course materials and 54 per cent wanted to keep the 
Student Supplement. 



Table 20 

Student Attitudes Regarding Retaining a 
Copy of the HFE Course Materials for Future Use 



Responses 



Percentage of Students 
Responding 



Yes No 



42% 58% 



Want to keep course materials? 
Want to keep supplement only? 



54 



46 



4. Course Presentation 



The percentage of student attitudes regarding the 
effectiveness of the course length (Table 21) was calculated 
with 52 per cent responding positively and 26 per cent neg- 
atively. Percentages of those regarding terminology used 
were 65 per cent positive and 9 per cent negative. Responses 
regarding participation in the course without the aid of an 
on-site instructor were relatively evenly distributed with 
37 per cent positive and 34 per cent negative. 

Regarding the latter set of responses, it is believed 
that contacting the CCS (who was not HFE trained) for ex- 
changing books and receiving tests and questionnaires may 
have been interpreted by students as contact with an on-site 
instructor. The CCSs' logs indicated that students seldomly, 
if ever, required their assistance with course content in 
the role of an on-site instructor. Additionally, this re- 
searcher received no communication from students asking for 
clarity of course instructions or content, even though it 
was encouraged in students' instructions. 

It must also be pointed out that 42 per cent of .the 
course participants desired to keep a copy of the course les- 
son materials, even though they had been informed that no 
assistance would be available upon completion of the evaluation 
period . 

Additional amplifying information regarding course 
presentation was extracted from written comments on the 



92 



Student Attitudes Regarding Effectiveness of 
Course Presentation 
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questionnaires and verbal comments recorded by the CCSs. 

These have been paraphrased or quoted directly when repre- 
sentative of the majority of the course participants: 

a. The level of detail presented throughout the 
course materials presented no difficulty in content compre- 
hension according to student reponses in the questionnaires. 
Some comments were made which indicated that students would 
like to see an even greater detail for some areas, however, 
they were not specific as to which areas. 

b. No negative comments were received regarding the 
course length. The time required for course participation 
was not overburdening to them. However, students felt that 
the format caused time to be wasted which could have been 
spent on the course content. 

c. The only feedback students received while taking 
the HFE course was provided in the answers at the end of each 
information frame. No periodic self-check tests were avail- 
able with the course materials. According to the course 
sponsors, none have as yet been developed. The sponsors had 
intended that measurement of the students ' attainment of the 
course LOs would not be necessary as their mastery of the 
content would become apparent as it was applied in the normal 
performance of their jobs in designing and implementing tests 
for evaluation of the human factors aspects of equipments and 
systems . 
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5 . Course Availability 

According to the course sponsors, original plans 
called for the U. S. Army to supply all necessary materials 
to its own personnel, while the cognizant Navy commands 
would fulfill Navy requests, once the course was finalized. 
However, present constraints on budget and personnel to make 
corrections and implement the computer program may hinder, 
if not prevent, this effort. 

6 . Course Maintainability 

Future plans of perceived needs in this area cannot 
be reported, as none were received from the sponsoring com- 
mands. However, even during this preliminary evaluation, its 
importance was»clearly noted. At present, course materials 
refer to specific pages of MIL-STDs and handbooks which 
caused some confusion to course participants when using MIL- 
STD 1472C. The HFE course makes references to specific 
pages within MIL-STD 1472B. In the latest version of the MIL- 
STD, the pages had been renumbered and students stated that 
there was some difficulty in finding the correct figure or 
information they were assigned to read. Additionally, the 
United States has converted their measuring system to metrics 
and this is not reflected within the course materials. 

7 . Course Supportability 

Future plans of perceived needs in this area cannot 
be reported as none were provided from the sponsoring commands. 
At present, supporting commands remain to be identified. 
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V. CONCLUSIONS AND RECOMMENDATIONS 



A. CONCLUSIONS 

The Human Factors Engineering Self-Paced Course has the 
potential to be most effective, based on its demonstrated 
ability to successfully provide the stimuli for the transfer 
of knowledge and skills to the Course Group in support of 
the intent of the course originators' task requirements and 
terminal objectives. The incorporation of the recommendations 
provided below would provide the course the potential to pass 
a validation attempt in accordance with CNTECHTRA requirements. 

Further validation of course content validity should in- 
elude a study of the Course Group participants after having 
had the opportunity to apply their newly gained knowledge 
and skills. in the performance of their jobs in designing and 
conducting human factors tests. 

B. RECOMMENDATIONS 

It is recommended that the course designers/sponsors 
comply with the standards and criteria identified by CNTECHTRA 
in NAVEDTRAs 110A and 106A for the development of military 
educational courses and incorporate the following changes into 
the Human Factors Engineering Self-Paced Course. 
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1 . Course Learning Objectives 

Rewrite and rearrange course terminal objectives and 
develop lesson and module enabling objectives. Incorporate 
the elements required for acceptable learning objectives which 
appear to be missing from those presently given for this 
course: (1) specific description of expected learner .action, 

(2) conditions under which the action will take place, and 

(3) standards or criterion which must be reached by the stu- 
dents. Evaluate them again according to the six criteria re- 
quired by NAVEDTRA 106A. The 17 course terminal learning 
objectives have been restructured (Table 22) to meet the cri- 
teria. The learning objectives should, however, be presented 
in the same order as the course content. A recommended re- 
ordering of the course’s original terminal objectives have 
been provided in Table 23. 

2. Course Design 

a. Sequence of Course Content. 

Reevaluate the sequence of the content as it is 
presently given to ensure that knowledge and concepts are 
taught before the students are required to apply theni, i.e., 
task analysis before human performance evaluation. This will 
support the ’’building block" concept of learning skills and 
provide the student a better opportunity to attain success 
and avoid unnecessary frustrations. 

b. Layout of Student Supplement and Lesson Books. 

Follow the guidelines provided in NAVEDTRA 110A 

which addresses the proper physical layout and required 
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Table 22 



Examples of Recommended Restructured Terminal 
Learning Objectives IAW NAVEDTRA 106A 



1. Define in writing common terms used in human factors 
engineering in accordance with lessons 1-40. 

2. Given a list of HFE reference documents, select the major 
topics addressed by each in accordance with lessons 1-40. 

3. Given an HFE problem, demonstrate the ability to resolve 
it by correctly selecting the HFE reference and locating 
within its contents the section which provides the ne- 
cessary information in accordance with lessons 1-40. 

4. Describe in writing the basic goals of HFE in a materiel 
development program in accordance with lessons 1-4. 

5. Describe in writing the inter-relationships of the four 
major factors of HFE in a materiel development program 
i.e., human performance requirements, personnel selection 
criteria, training and equipment design- in accordance 
with lessons 1-4 and 28-29. 

6. Given the stages of the basic systems acquisition cycle, 
describe in your own words the level of involvement an 
HFE designer and evaluator should have at each point. 
Include in your discussion the purpose of your testing 
and the basic steps you would follow to become involved 
at each stage. Responses should be in accordance with 
lessons 21 and 22. 

7. Discuss in writing the importance and purpose of evalua- 
ting human factors in a systems -concept rather than just' 
evaluating the operator or the machine individually in 
accordance with lessons 2, 21, and 22. 

8. Given an HFE proglem, analyze systems concept employing 
task analysis and/or determine the human factors require- 
ments which must be considered when designing the piece 
of equipment in accordance with lessons 6, 7, 14, and 
21-24. 

9. Given an HFE problem, identify 3 factors and 3 forces 
which would affect human performance and describe how you 
would measure each, i.e., what equipment to use and what 
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unit of measure is appropriate, in accordance with lessons 
9, 10, and 15-19. 

10. Discuss in writing the differences between HFE measure- 
ments taken in the field and those later in a laboratory- 
setting. Include in your discussion the potential dif- 
ficulties of applying generalized laboratory data to 
field situations, in accordance with lessons 31 and 32. 

11. Discuss in writing the importance of performing testing 
both in a controlled environment and that in the "real 
world". Responses are to be in accordance with lessons 
31 and 32. 

12. Given a list, identify the "experimental control" measures 
necessary in any test involving the human performance and 
describe the probable effects on the test data if they 
are absent, in accordance with lessons 31 and 32. 

13. Given an HFE problem with human performance data available 
calculate system performance reliability and assess the 
potential effectiveness of the system. Round your answer 
to the nearest thousandths. Responses are to be in 
accordance with lessons 21, 22, 26, 33 and 34. 

14. Given three different examples of a task analysis for the 
same piece of equipment, select the one which is done 
correctly, in accordance with lessons 23 and 24. 

15.. Given an HFE problem, determine performance measures for 
• the dependent variables of time and. error in accordance 
with lessons 33 and 34. 

16. Describe the relationship between human factors engineer- 
ing and the engineering specialties of reliability, 
maintainability and safety in accordance with lessons 
25-27. 

17. Given an HFE problem, state the HFE standards and speci- 
fications necessary to evaluate human performance in ac- 
cordance with the guidelines and reference documents 
presented in lessons 21 and 22. 

18. Interpret the data given in an analysis of variance table 
performance scores from an experiment having three inde- 
pendent variables with one statistically significant 
first order interaction, in accordance with lessons 33 
and 34. 
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Table 23 



Recommended Resequencing of the Human Factors Engineering 
Self-Paced Course Terminal Learning Objectives 

Upon completion of the Human Factors Engineering Course, the 

student will demonstrate the following capabilities and 

knowledge : 

1. An understanding of common terms used in human factors 
engineering . 

2. A familiarity with human factors references and an ability 
to use them. 

3. An awareness of potential sources of technical information 
on major human factors topics. 

17. An ability to interpret the standards and specifications 
of the human factors engineering community, through the 
use of HFE references. 

4. An understanding of the goals of human factors engineering 
in a materiel development program. 

5. A familiarity with the acquisition cycle. 

7. An understanding of the kinds of factors and forces which 
affect human performance and an ability to identify them. 

15. A familiarity with task analysis. 

7. An ability to measure the factors and forces which affect 
human performance. 

10; An understanding of basic statistical techniques, such as 
an analysis of variance. 

12. An ability to formulate performance measures for the de- 
pendent variables of time and error. 

11. An ability to calculate human performa-nce reliability. 

5. An ability to integrate human factors principles in a 
DOD sponsored program. 

14. An understanding of the major techniques used by human 
factors specialist during system synthesis, design, and 
development . 
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6. An ability to determine human performance requirements 
in a systems concept. 

13. An ability to analyze human performance data within the 

context of "system effectiveness" and "system reliability". 

16. An awareness of the relationship between human factors 
engineering and the engineering specialist of reliability, 
maintainability and safety. 

8. An awareness of the differences between field and labora- 
tory measurements . 

9. An awareness of what "experimental control" measures are 
necessary for any test involving human performance and 
the effects in their absence. 

17. An ability to apply the standards and specifications of 
the human factors engineering community. 
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elements of module (lesson) booklets and lesson topics: 

(1) Instructional materials are required to have 
a specific numbering system which clearly shows the relation- 
ships between terminal objectives, enabling objectives and 
course content. 

(2) Each module booklet is required to have a 
cover page, module overview and lesson topics. Samples of 
the module overview and cover page are provided in Appendix I. 

(3) Lesson topics cover, wholly or in part, 
terminal objectives. Progress checks (self- tests) are pro- 
vided for the student at the end of lesson topics to measure 
their ability to achieve the terminal objectives and determine 
the need for remediation. 

(4) Each lesson topic shall contain the items 
listed below: 

Lesson Topic Cover Page* 

Lesson Topic Overview* 

List of Study Resources 
Lesson Topic Summary 
Narrative Form of Lesson Topic 
Programmed Instruction Material 
Lesson Topic Progress Check, complete with 
feedback and remediation. 

Samples of those items asterisk'd above are provided in 
Appendix J. 
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Format of Course Lesson Books 



c . 

(1) Review "information frames" and questions to 
ensure that sufficient information has been provided to answer 
the questions before they are asked, i.e., Lesson 15, page 84, 
"frame from page 80"; Lesson 17, page 68, "frame from page 67". 

(2) Review "wrong answer frames", increase pro- 
fessionalism in replies and provide reinforcement by directing 
the student to a specific area to help choose the correct 
answer on the next try, i.e.. Lesson 12, page 68 "(3) We 
know you don’t know, but we want you to give it a shot. Re- 
turn to page 39...". 

(3) Review the quality of the question asked. 
Rewrite those which are ambiguous, require guessing or really 
offer no doubt as to the answer because of the examples pro- 
vided in the wording of the answer choices, i.e.. Lesson 13, 
page 96, "frame from page 93". 

(4) Review the story line using the ficticious 
Lt, I. M. Eager and if it. is decided to maintain the story 
line, upgrade the examples and remove extraneous portions 
which do not add to the content of the course. Provide more 
"real world" examples which could add greater meaning and 
clarity to the content. 

(5) Change the "page flipping" format of the 
written version of the course. The U. S. Navy correspondence 
courses provide an acceptable format and one with which some 
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military personnel are already familiar. Another popular for- 
mat is used in the Elementary Algebra: Lecture Lab by Arthur 
H. Heywood. 

3 . Course Content . 

a. Review course content to be sure that the tasks 
identified in the learning objectives are adequately addressed, 
i.e., calculation of reliability. 

b. Correct clerical errors, i.e., Table 33.5 of 
Student Supplement; Lesson 20, page 74, says ’’frame from page 
71" when it should read "..-.page 24”; Lesson 19, page 91, frame 
from page 26: reads "By finding the point of ... between four 
hours...", should read "...two hours". 

4 . Course Presentation . 

a. Provide student self-check tests following each 
lesson topic and following each module ivith a final comprehen- 
sive test upon course completion (after the course review). 

The questions at the end of each information frame don't allow 
the student to adequately evaluate their understanding or 
acquisition of the necessary concepts and skills. Include 
reinforcement for end-of-information-frame tests by referring 
the student to the appropriate sections for review based on 
students’ incorrect answers. 

b. Provide course tests so that students can judge 
how well they had attained the course learning objectives be- 
fore they attempt to apply it in their normal jobs. This will 
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allow them to go back and review the areas in which they are 
weak. A certificate of completion and credit can then be 
awarded upon satisfactory completion of the course. Comple- 
tion of the course may also be made a part of each command’s 
qualification procedures. 

5 . Course Availability, Maintainability, and Supportabil - 

ity . 

a. Provide plans and estimates required for future 
implementation of the HFE Course and an outline of the ne- 
cessary support in those areas as specified in NAVEDTRA 110A. 

b. Update this version of the course to be consistent 
with MIL-STD 1472C vice MIL -STD 1472B. When referencing 
material within any of the reference documents used within 
the course, do not use page numbers; provide section numbers 
instead. Addtionally, make references more specific when 
dealing with MIL-STD documents; do hot just refer to the 
document as a whole when addressing a specific section. 

c. Recommend a schedule of periodic course review 
and updates as indicated by NAVEDTRA 110A. 



I 
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VI . SUMMARY 



As this study indicates, the Human Factors Engineering 
Self-Paced Course has the strong potential for providing 
the stimuli to transfer knowledge and skills to its students 
Whether this course provides sufficient content to effective 
ly bridge the gap between the military technical documents 
and standards to allow for their increased usage by evalu- 
ators on the job is an area for further study. 

There is an interesting side note relative to the low 
level of human factors familiarity by all experiment parti- 
cipants as indicated by the results on the pretest: could 
this possibly be substantiation to the claim made by GAO in 
their report to the U. S. Congress [Ref. 1: pp. 1-27] which 
has been referenced periodically throughout this thesis? 
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APPENDIX A. 



RESOURCE DOCUMENTS SUMMARIES 

A brief summary of the various military standards, hand- 
books, and documents required for use by the Human Factors 
Engineering Self-Paced Course participants is provided 
within this Appendix: 

1. MIL-H-46855B 

2. MIL-STD- 1472C 

3. MIL-STD-1474B 

4. MI L-HDBK- 759 

5. Army Regulation 602-1 

6. Technical Memorandum 29-76 

Additonally, summaries or copies of the abstracts are 
provided for documents to which the Human Factors Engineering 
Self-Paced Course referred, but are not essential for course 
completion : 

1. MIL-STD- 721B 

2. FED-STD-595 

3. Guide to Human Factors Engineering General Purpose 
Test Planning (GPTP) 

4 . HFTEMAN 

5. HEDGE 

6 . MOAT 
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REQUIRED DOCUMENTS 



1. MIL-H-46855B Human Engineering Requirements for Military 

Systems, Equipments and Facilities 

"MIL-H-46855B establishes and defines the overall require- 
ments for applying the human engineering priciples and cri- 
teria presented in MIL-STD-1472C during the procurement of 
military systems, thereby effectively integrating man into 
the system. The specification requires the prospective con- 
tractor to state his approach in his Human Engineering Pro- 
gram Plan which is submitted in response to the Request for 
Proposal (RFP) . The Human Engineering Program Plan, upon 
acceptance of hte proposal, becomes part of the procurement 
contract . 

The three major areas of the system acquisition process 
where these human factors engineering principles are to be 
employed are analysis, design and development, and test and 
evaluation. The objectives of analysis are identification 
and definition of system operations, maintenance, training 
and control functions; allocation of these functions to man 
and/or machine; analysis of the tasks comprising these func- 
tions; and development of system specific human engineering 
design criteria and operation and maintenance procedures. 

The human engineering inputs developed in the analysis phase 
and comensurate with MIL-STD-1472C are to be incorporated 
into the detailed design. These human engineering provisions 
shall .be evaluated during the design reviews to ensure their 
adequacy. The purpose of the test and evaluation phase is 
assurance of the fulfillment of MIL-H-46855B and contract 
requirement, demonstration of conformance to MIL-STD-1472C 
quantification of man-machine system performance, and indi- 
cation of the introduction of possible undesirable design or 
procedural features. 

’MIL-STD-1472C and MIL-H-46855B are complementary direc- 
tives that deal with requirements for implementation of human 
factors engineering principles in the development and acqui- 
sition of military systems, equipment and facilities. The 
former establishes the criteria by which the systems are to 
be human engineered, and the latter establishes the require- 
ments for maintenance and maintainability as integral parts 
of the human engineering of the total system." 
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2. MIL-STD-1472C. Human Engineering Design Criteria for 

Military Systems, Equipments and Facilities , 

"This standard establishes general human engineering cri- 
teria for development of military systems, subsystems, equip- 
ment and facilities. 

The purpose of this standard is to present human engineering 
design criteria, principles, and practices to achieve mission 
success through integration of the human into the system, 
subsystem, equipment, and facility, and achieve effectiveness, 
simplicity, efficiency, reliability, and safety of system 
operation, training, and maintenance. 

More specifically, its purpose is to present human engineer- 
ing design, criteria, priciples and practices to be applied in 
the design of systems, equipment and facilities so as to: 

a. Achieve required performance by operator, control and 
maintenance personnel. 

b. Minimize skill and personnel requirements and training 
time . 

c. Achieve required reliability of personnel-equipment 
combinations . 

d. Foster design standardization within and among systems. 

The standard includes a compilation of anthropometric data 
from several military sources (MIL-STD- 721B , MIL- STD- 1474B 
and FED-STD- 595) , an extensive collection of control/display 
design criteria, hazard and safety considerations, and require- 
ments for certain specialized systems which may also be applied 
to such equipments as ground vehicles, remote handling devices, 
air and ship crew stations." 



3. MIL-STD-1474B Noise Limits for Army Materiel 

"Three distinctly different types of 'noise criteria' 
which are used to limit noise exposure have evolved over the 
years. It is important to distinguish among the three types 
in order that the proper type may be chosen for application and 
use in various situations. The three types of noise criteria 
are : 

a. Hearing damage-risk criteria 

b. Hearing conservative criteria 

c. Materiel design standards 

« 3 

This document is a design standard for noise. It is based 
on provisions of TB-MED-2S1 with respect to noise exposure 
criteria. MIL-STD-1472C extracts communications criteria from 
it . " 
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4. MIL-HDBK- 7 59 . Human Factors Engineering Design for Army 

Materiel . 

"This • handbook gives the design engineer both human factors 
engineering design priciples and detailed criteria. The design 
principles are stated as general rules to be applied during 
system-development programs or .as essential items that must be 
considered during design to insure that- sound human factors 
engineering practices will be incorporated. The detailed cri- 
teria consist of dimensions, ranges, tolerances, and other 
specific data. In some cases, the range of acceptable dimen- 
sions and other factors may be rather large. Where only the 
minimum and maximum are given, design engineers may select any 
part or item within the recommended range. But where optimum 
dimensions are given, designers should aim to approximate them 
whenever possible. 

The purpose of this document is to establish in handbook 
form general data and detailed criteria for human factors 
engineering application in the design and development of army 
materiel . 

The information in this handbook is a consolidation of 
the material contained in four U. S. Army Human Engineering 
Laboratory Standard Documents; HEL S-2-64A, S-3-65, S-6-66, 
and S-7-68.. The user will also note the similarities of this 
document with MIL-STD-1472C. " 



5. Army Regulation 602-1. Personnel -Materiel Systems : Human 

Factors Engineering ' Program . Washington, D. C., June 1976 . 

"This regulation prescribes policies and procedures and 
assigns .responsibilities for human factors engineering (HFE) 
in the Department of the Army. For the purpose of this reg- 
ulation, -HFE is defined as a . comprehens ive technical effort 
to integrate all personnel characteristics (skills, training 
implications, behavioral reactions, human performance, 
anthropometric data and biomedical factors) into Army doctrine 
and systems to assure operational effectiveness, safety, and 
freedom from health hazards." 



6. TECHNICAL MEMORANDUM 29-76. Guide for Obtaining and An - 
alyzing Human Performance Data in a Materiel Development 
Pro] ect ♦ 

"The specific objectives of this report are to: (1) de- 

scribe how to conduct and report an HFE test according to the 
requirements of DI-H-1334A, (2) detail the expenditures in 
time and money associated with the conduct of an HFE test, 
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(3) provide examples of HFE test reports for systems in "ex- 
perimental prototype" and "advanced development" phases of 
development, (4) describe the uses of the obtained HFE test 
data as a function of system development, and (5) explain the 
impact of the DI-H-1334A findings on a materiel development 
program. 

This document is written for government contract monitors, 
contract project directors, and contractor HFE personnel. 

The guidelines for conducting and reporting on the HFE test 'are 
intended for experienced HFE personnel. 

Questions of what data are to be collected, how they are to 
be collected, and how the data can be used are discussed in 
Chapters 2, 3, and 4 of this report. Chapter 2 is a guideline 
for planning, conducting, analyzing, and reporting on an HFE 
evaluation according to the requirements of DI-H-1334A. Pro- 
cedures for managing the HFE evaluation, allocating test 
personnel* developing test cost estimates, etc., are also con- 
tained in Chapter 2. This information will aid project man- 
agers in the administration and organization of an HFE 
evaluation. The explanations of the DI -H-1334A- requirements 
will assist the contract monitor to understand and monitor HFE 
tests and ensure that all of the requirements of DI-H-1334A are 
satisfied. 

Chapters 3 and 4 supplement the guidelines given in Chapter 
2 by giving detailed examples of HFE reports, written according 
to the requirements of DI-H-1334A. Chapter 3 presents the 
HFE test report of a system. in the experimental prototype 
stage of development. This sample HFE report focuses on de- 
terminining the feasibility of human performance, the appro- 
priateness of the tasks allocated to the operator and to the 
machine, and the adequacy of the workspace layout. This report 
also demonstrates procedures for conducting HFE tests and mock- 
up equipment. 

Chapter 4 contains the HFE test report of an advanced 
development prototype. The emphasis in this stage of develop- 
ment is on determining the capability of the operator to 
perform his assigned tasks within his prescribed time and 
error standards. This test report also evaluates the adequacy 
of operator selection and training, as well as the equipment 
configuration. 

Chapter 5 discusses implications of human performance tests. 
The uses to which data can be applied and the problem associ- 
ated with conducting HFE tests are also described. By de- 
scribing how HFE test data can be applied and the benefits 
of collecting the data, program managers can better appreciate 
the need for HFE testing." 
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RECOMMENDED DOCUMENTS 



1. MIL-STD-721B Definitions of Effectiveness Terms for 
Reliability, Maintainability, Human Factors and Safety 

"Applying effectiveness terminology within the Department 
of Defense and industry requires good communications and 
coordination which, in turn, requires common definitions and 
terminology. This Standard lists and defines words and terms 
most commonly used in reliability, maintainability, human 
factors and safety. Statistical and mathematical terms which 
have gained wide acceptance are not defined in this Standard 
since they are included in other documents, e.g., MIL-HDBK- 217 . 

The following criteria were used for the inclusion of 
terms and definitions deemed pertinent to the scope of this 
Standard: 

a. Terms and their definitions which are: 

(1) Important in procurement of weapon systems for 
precise definition of effectiveness criteria. 

(2) Unique in their definition, allowing no other 

meaning. 

(3) Expressed clearly, preferably without mathematical 

symbols. 



(4.) Fully explanatory, without reference to any other 
source documents. 

b. Terms that were avoided include: 

(1) Those found in ordinary technical, statistical, 
or standard dictionary or text having a singularly acceptable 
meaning when used in the context. 

(2) Terms already existing in other Military Standards 
outside of the project scope. 

(3) Multiple word terms, unless needed for uniqueness. 

c. The purpose of this Military Standard is to standardize 
on meanings of terms for the particular application, not to 
compile a handbook. 



I 
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2. FED STD 595 Colors 



"The purpose of this standard is to present in covenient 
form a collection of standard colors currently used by the 
various departments of the Government. These standard colors 
identified by 5-digit numbers, are defined by fundamental 
colorimetric data. For reference purposes, each color is 
reproduced within a 1/2 x 1 inch chip. For procurement, in- 
spection and other color matching purposes, 3x5 inch color 
chips are issued as specific standards. These chips are 
identified by the same 5-digit numbers. 

Spectrophotometric curves and Commission Internationale 
de l'Eclairage (CIE) data for each chip are included as 
Volume II to this standard. These are to be used as basic 
standards for reproducing future issues of the chips, also 
for determining changes which may, or may not, have occurred 
in chips in stock. The spectrophotometric curves and CIE 
measurements may also be used for acceptance testing purposes 
in lieu of the 3x5 inch chips if so specified. However, 
accurate comparison can be made only if values and curves 
are obtained on the same instrument standardized under the 
same conditions." 



3 . Guide to Human Factors Engineering General Purpose Test 

Planning (GPTP ] 

"This report is concerned with human factors engineering 
test and evaluation program planning, which is generalizable 
across naval weapons systems except for nuclear weapons and 
propulsion subsystems. The Test and Evaluation Program 
Planning Guide is prepared for use by test planners and' is in 
accordance with DOD Directive 5000.3, NAVMATINST 3900.0, 
NAVMATINST 3960.6, OPNAVINST 3960.10, and OPNAVINST 4720.9. 
The human factors engineering Test and Evaluation Program 
Plan specifically addresses developmental and operational 
tests as well as production acceptance tests to provide data 
for satisfying the Defense System Acquisition Review Council 
milestones for system acquisition." 



4. HFTEMAN Human Factors Test § Evaluation Manual, Vols. I, 
II, and IT! 

"In most military systems the ultimate effectiveness of 
the system depends on the capability of the human to operate, 
maintain, transport, erect, or otherwise use the system equip 
ment . 
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In the test and evaluation of such systems adequate con- 
sideration must be given to the human element. 

Human Factors Test § Evaluation Manual was developed to 
assist Navy and Marine Corps Test and Evaluation personnel in 
the evaluation of the human factors aspects of equipment items. 

Human Factors Test § Evaluation Manual is primarily di- 
rected toward the Human Factors Engineering (HFE) . The detail 
provided' for the Test and Evaluation Plan: 

-- Guidelines concerning what to evaluate. 

-- Standards or criteria to evaluate against. 

-- Planning information on how to design, setup, conduct, 
and analyze data from a human factors enginering test 
or evaluation. 

Human Factors Test 5 Evaluation Manual is therefore used 
when the detail Test and Evaluation plan is being generated. 

It enables the planner to identify HFE: 

-- Test objectives. 

-- Test methods, procedures and conditions. 

— Test measurements to be acquired and recorded during 
the test or evaluation. 

-- Criteria or standards against which measurements are 
compared . 

-- Test data collection materials such as questionnaires 
or checklists. 

-- Test data analysis, presentation and reporting require- 
ments . 

Human Factors Test § Evaluation Manual consists of three 
volumes : 

-- Volume I - Data Guide: contains the guidelines concern- 
ing what to evaluate for different classes 
of equipment and types of tests. • 

-- Volume II- Support Data: contains additional criteria 
expanding on the guidelines contained in 
Volume I. 

-- Volume I II -Methods and Procedures: contains guidance on 
how to design, set up, conduct and analyze 
data obtained in HFE Test 5 Evaluation program. 
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5 . Human Factors Engineering Data Guide for Evaluation (HEDGE ) 

"The purpose of the information in HEDGE is to enable you 
to expand your test capabilities in considering the human 
element. They will provide you with a strategy for viewing 
any item which is undergoing testing from the standpoint of 
the soldier who must ultimately operate, maintain, or other- 
wise utilize it. The use of these materials, in addition to 
standard Task and Design Checklists and Questionnaires, will 
enable you to tailor your HFE subtest to a specific item. 

These materials have been prepared especially for you: 

a. They are intended to support test engineers not design 
engineers . 

b. They were designed with your specific tasks in mind, 
i.e., preparing a Test Plan, conducting a test, analyzing 
and interpreting test data, and generating the test report. 

Because these materials offer you a strategy for conducting 
human factors testing rather than a compendium of facts, the 
results which you obtain will be directly proportional to 
your intelligent and common sense application of the data 
presented . " 



6 . Mission Operability Assessment Technique: A System 

Evaluation Methodology (MOAT ) 

"The MOAT is an evaluation methodology that measures the 
operability of a system or subsystem in terms of operator 
tasks performed during a mission. MOAT addresses the problem 
of how well an operator can use a system or subsystem to per- 
form tasks within the mission context. Contrasted to evalu- 
ations using human engineering design criteria which present 
only pass or fail information, this technique provides infor- 
mation on the degree of system and/or subsystem success or 
failure. This report discusses how MOAT was developed, how 
MOAT can be used on existing or emerging systems, and how MOAT 
will be expanded in future development to include multicrew 
station evaluations. This report examines how the three under- 
lying technologies have been integrated into one comprehen- 
sive methodology. Task analysis, scaling methodology, and 
multi-attribute utility theory are* discussed in terms of MOAT 
development and application. Finally, the report discusses 
the importance of developing systems evaluation methodologies 
which provide decision makers with meaningful information 
necessary for effective decision making." 



APPENDIX B. 



HUMAN FACTORS ENGINEERING SELF-PACED COURSE 
TASK REQUIREMENTS 

Upon completion of the HFE Course, students should know: 

1. The meaning of common terms used in human factors engi- 
neering. 

2. The principal HFE references (regulations, standards, 
specification, guides, etc.) and where ' in them to look 
for the answers to specific types of HFE problems. 

3. Potential sources of technical information on major topics 
within human factors engineering. 

4. The goals of HFE in a materiel development program (i.e., 
the compatibility of the four factors: human performance 
requirements, personnel selection criteria, training and 
equipment design). 

5. When, how, and for what purpose in the general scheme by 
which the Department of Defense (DOD) develops materiel 
human factors engineering activities should be performed. 

6. How to determine the human performance requirements in a 
systems concept. 

7. The kinds of factors and forces which can affect human 
performance and how to identify and measure them. 

8. The differences between measurements taken in field vs. 
•laboratory settings, and the difficulties of generalizing 

laboratory data to field situations. 

9. How to formulate and then state in comprehensible English 
performance measures applicable to a specific system for 
the general dependent variables of time and error. 

10. What "experimental control" measures are necessary in any 
test involving human performance and the probable effects 
on test data if they are absent. 

11. How to analyze human performance data (e.g., time and 
error) within the context of "system effectiveness" and 
"System reliability". 
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12. How to calculate human performance reliability. 

13. The major techniques used by human factors engineers 
during system sythesis, design and development. 

14. How to recognize a good task analysis. 

15. The relationship between human factors engineering and 
the engineering specialities of reliability, maintain- 
ability and safety. 

16. How to interpret and apply the .standards and specifica- 
tions needed for human factors engineering that deal 
with human performance. 

17. The correct method of stating criteria for a vehicle 
which must be inaudible 100 meters away from it, and 
for an alarm which must be audible for 500 meters. 

18. How to determine the minimum height of letters on a 
sign which must be visible at 100 meters on a sunny day. 

19. How to interpret an analysis of variance table of per- 
formance scores in an experiment having three indepen- 
dent variables with one statistically significant main 
effect and one statistically significant first-order 
interaction. 
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APPENDIX C. 



HUMAN FACTORS ENGINEERING SELF-PACED COURSE 
TERMINAL LEARNING OBJECTIVES 



Upon completion of the Human Factors Engineering Self-Paced 

Course, the student will demonstrate the following capabili- 
ties and knowledge:. 

1. An understanding of common terms used in human factors 
engineering. 

2. A familiarity with human factors references and an ability 
to use them. 

3. An awareness of potential sources of technical information 
on major human factors topics. 

4. An understanding of the goals of human factros engineering 
in a materiel development program. 

5. An ability to integrate human factors priciples in a DOD 
sponsored program. 

6. An ability to determine human performance requirements 
in a systems concept. 

7. An understanding of the kinds of factors and forces which 
affect human performance and an ability to identify and 
measure them. 

8. An awareness of the differences between field and labora- 
tory measurements. 

9. An awareness of what "experimental control" measures are 
necessary for any test involving human performance and 
the effects in their absence. 

10. An understanding of basic statistical techniques, such 
as an analysis of variance. 

11. An ability to calculate human performance reliability. 

12. An ability to formulate performance measures for the de- 
pendent variables of time and error. 

13. An ability to analyze human performance data within the 
context of "system effectiveness" and "system reliability". 
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14. An understanding of the major techniques used by human 
factors specialists during system synthesis, design, and 
development. 

15. A familiarity with task analysis. 

16. An awareness of the relationship between human factors 
engineering and the engineering specialists of reliability, 
maintainability, and safety. 

17. An ability to interpret and apply the standards and speci- 
fications of the human factors engineering community. 
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APPENDIX D. 



HUMAN FACTORS ENGINEERING SELF-PACED COURSE 
LESSON TOPICS OUTLINES6 



Lesson 1 : Welcome to Human Factors Engineering 

I. Course Introduction 

II. Importance of Human Factors 



Lesson 2 : Why Human Factors? 

I . Story Line 

A. Introduction to main characters 

1 . Lt . I . M. Eager 

2. Capt. B. Smart 

II. Human Factors Engineering 

A. Definition of 

B. Historical Perspective 

III. Systems 

A. Man-machine Systems 

B. Components of Man-machine Systems 



Lesson 3: Tragic Mistakes and Positive Consequences 

I. Historical Perspective 
A. Examples 

II. Common Errors - Reading and Interpreting Instruments 

III. Common Errors - Operating Controls 

IV. Current Status of Human Error 

V. Man-Machine Incompatibilities 

A. Man-machine Capability Differences 



^Extracted directly from the Student Supplement of the 
Human Factors Engineering Self-Paced Course. 
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Lesson 4 : Basic Information Processing, or Is Man a Machine? 

I. Attention 

A. Limits to 

B. Selective Attention 

II. Judgments 

A. Absolute 

B. Relative 

III. Man-machine Systems. 

A. Man the Processor 

B. Stimulus-Response Codes 

C. Information Transmission 

D. Input-output Processes 



Lesson 5: History and Related Technology, or Human Factors, 

This Is Your Life 

I. History - HFE 

A. Ancient Cultures 

B. Industrial Revolution 

C. World War II 

D. Current Efforts 

0 

II. Stage of HFE Development 

A. Pretechnology 

B. Aerospace ’ • 

C. Sociotechnical 

D. Cosmopolitan 

III. Review 



Lesson 6 : Anthropometry, or Do I Fit? 

I. Introduction- 

A. Definition 

B. Historical Perspective 

II. Design Principles 

A. Adjustability 

B. 5th - 95th Percentile Range 
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Lesson 7 : Work Space Design and Arrangement, or Don't Cramp 

My Style 

I. Work Space Envelope 

A. General Design Principles 

II. Analytical Methods 

A. Indexing/Rating Method 

B. Link Analysis 

C. Prototypes 

III. Component Location and Spacing Problems 



Lesson 8: Vision Capabilities, or A Shot In The Dark! 



I. 


Visual Anatomy-Supplement 




II. 


Visibility 




III. 


Panel Lighting 

A. Flood Lighting 

B. Integral Lighting 




IV. 


Visual Detection, Identification, and Estimation 

• 


Lesson 


9: Vision Displays, or Are 


My Eyes Deceiving Me? 


I. 


Types of' Displays 

A. Quantitative 

B. Qualitative 

C. Static 

D. Dynamic 




II . 


Scales 

A. Digital 

B. Fixed Pointer 

C. Moving Pointer 




Ill . 


Signal and Warning Lights 
A. Detection Factors 




IV. 


Alphanumeric and Symbolic 


Characters 
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LESSON 10 : Auditory Presentations, or When Is An Alarm Not 

' An Alarm? 

I. General Types of Displays 

A. Displays Using Auditory Modalities 

B. Nature of the Message 

II. Sound Characteristics 

A. dB, Hz. 

B. Frequency, Intensity 

C. Human Reception Capabilities 

III. General Display Principles 

IV. Presentation Principles 

V. Warning and Alarm Systems 



LESSON 11 : Standardization of Controls, or Which Way Is Up? 

I. Standard Arrangement - Population Stereotypes 

II. Categorizing Controls 

A. Quantitative, Qualitative, Representational, and 
Continuous 

III. Functions of Controls 

A. Continuous Adjustment Settings 

B. Discrete Settings 

IV. Types of Controls 

A. General Types - Linear, Rotary 

B. Specific - Pushbutton, Toggle Switch, Knobs, etc. 

V. General Rules for Selection 

A. MIL-STD- 1472C and MIL-HDBK-7S9 



LESSON 1 2 : .Positioning of Controls, or The Right Place At The R 
Right Time 

I. Accidental Activation of Controls 
A. Prevention Methods 
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II. Location of Controls and Displays 

A. Priority, grouping association factors 

B. Location- and body position 

C. Spacing between controls 



LESSON 13 : Control Dynamics, or How Hard To Twist The Knob 

I. Design Concepts 

A. S-R compatibility 

■B. Control -display ratio, a type of compatibility 

II. Types of resistance 

A. Static friction 

B. Sliding friction 

C. Elastic resistance 

D. Viscous damping 

III. Feedback 

A. Intrinsic 

B. Extrinsic 



LESSON 14 : Other Senses, or Controls That Have Shaped-Up 

I. Review lessons 11-13 
II . Touch 

A. Pressure Sensitivity 

B. Pain 

C. Temperature 

III. Touch Coding 

A. Shape coding 

B. Class 'A' and 'B' design 

C. Texture coding 



LESSON 15 : Vibration and Acceleration, or Take It Easy I Have 

Weak Stomach! 

I. Vibration 

A. Definitions 

B. Measurement 

C. Body Parts affected 

II. Whole Body Vibrations 

A. Performance effects 

B. Tolerable limits 
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III. Acceleration 

A. Performance effects 
C. Protective measures 



LESSON 16 : Vigilance, or Stay Awake If You Can 

I. Vigilance 

A. Definition 

B. Examples 

II. Performance Decrements Due to Vigilance 

A. Time Frame 

B. Magnitude of decrement 

C. Display characteristics 

III. Signal Characteristics 

A. Noise 

B. Signal intensity 

C. Rate of signal presentation 

IV. Other Factors Affecting Watch-keeping Behavior 

A. Environmental conditions, noise 

B. Atmospheric temperature 

C. Procedural conditions, work/rest schedules 

LESSON 17 : Temperature Effects, or Baby, It’s Cold Outside 

I. Reactions to Temperature Change 

A. Physiological 

B. Adaptation, sensitization and habituation 

C. Effective temperature 

II. Extreme Cold 

A. Physiological and performance effects 



LESSON 18: Atmospheric Effects or I Can't Breathe 



I. 


Description of Atmosphere 




A. 


Gases, density, pressure 


II. 


Hypoxia 




A. 


Factors causing 




B. 


Effects - physiological and performance 


III. 


C0 9 


and CO 




A. 2 


Physiological and performance effects 




B. 


Military specif iciat ions 
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IV. Radiation 

A. Measurement Definitions 

B. Effects of overexposure 

V. Prevention 

A. Mask 

B. Clothing 



LESSON 19 : Noise, or Can You Hear Me? 

I. Auditory Anatomy 

A. Structure 

B. How sound travels in the ear 

C. Mechanical and electrical transmission 

II. Properties of Auditory Stimuli 

A. Frequency, intensity 

III. Communication 
A. Maskers 



IV. Physiological Effects 

A. TTS 

B. PTS 

C. Damage risk criteria 

V. Protection Against Noise 



LESSON 20: Review of Lessons 1-19 



LESSON 21: System Acquisition 



I. Department of Defense Acquisition Policy 

A. Variation across systems 

B. 0MB A-109 

C. DOD 5000.1 



II. Phases of Weapons Systems Acquisition Process 

A. Concept exploration 

B. Demonstration and validation 

C. Full-scale development 

D. Production and deployment 



III. HFE and the Acquisition Process 
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LESSON 22: System Analysis, or The Big Picture 



I. Purpose of System Analysis 

A. Scheduling 

B. Identifying limiting factors 

C. System performance criteria 

D. Design Options 

E. Evalutation of Systems 

II. Stages of System Analysis 

A. Requirements analysis 

B. Function analysis 

C. Task analysis 

III. Major Problem Areas 

A. Subsystems' interaction 

1. Sectionalization technique 

B. Criterion determination 

C. Defining human performance effectiveness 



LESSON 23 : Task Analysis: History and Perspectives 

I. Recent History of Task 



II. Definition of Task Analysis 

III. Other Improtant Task Analysis Factors: 

A. Systems mission and function 

B. Job, task, sub-task, task element 

C. Task inventory, task taxonomy 



IV. Output of Task Analysis 

A. Design 

B. Training 

C. Test and evaluation 

D. Manning 

E. Workload 



LESSON 24 : Task Analysis, or Fitting Task Analysis into 

System 

I. Stages in Task Analysis Process 

A. Identify task, sub-task, task element 

B. Develop specific behavioral objectives 

C. Identify supporting skills and knowledge 



II. Task Statements 

A. Specific behavioral objectives 



the 
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III. 



Task Analysis Worksheet 



IV. Sampling Techniques 

A. Activity sampling 

B. Process Analysis 



LESSON 25 : Affordability, or When Can We Trade Off What? 

I. Trade-off analysis 

A. Definition 

B. Types: 

1. Geometry of design 

2. Manpower allocation 

II. General System Analysis 

A. Operational requirement 

B. Hardware design 

C. Manpower and training requirements 

D. Saftey, reliability , and other factors 

III. Four Major Steps in Trade-off Analysis 
A. Baseline Alternative 

1. ROC 

2. Initial Hardware 

3. Manpower and training requirements 

IV. Life-cycle Costs 

A. Computer models 



LESSON 26 : Maintainability, or Can Anybody Fix This? 

I. Basic Concepts 

A. Maintainability 

B. Maintenance 

C. Reliability 

D. Human performance 

II. Design Features 

III. Skill Application 

A. Capabilities/limitations 

B. Training 

IV. Predictions 
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LESSON 27 : Hazard Analysis, or A Stitch in Time 



I. Introduction 

A. HFE and safety 

B. HPE and safety design 

II. Introduction to Hazard Analysis 

III. Identification Phase 

A. Checklists 

B. Historical records - intermediate indicators 

C. General Investigations 

IV. Evaluation Phase 

A. Grouping according to category 

B. Ranking within category 

V. Cost Countermeasures 



LESSON 28 : Training the Right People 

I. Selection and Training in Design/Conceptual Phase 
A. Approaches 

1. Organizational Analysis 

2. Job analysis 

3. Person analysis 

II. Training Techniques 

A. On-the-job training 

B. Classroom training 

C. Computer-aided instruction 

D. Team Training 



LESSON 29: Does Training Work ? 



I. ISD Phases 

A. Analysis 

B. Design 

C. Develop 

D. Implement 

E. Control 



II. Validity and Reliability 
A. Types of validity 

1. Training 

2. Performance 

3. Intra-organizational 

4. Inter-organizational 



130 



III. Simulator Training Issues 



LESSON 30 : Psychophysical Methods, or Do I Detect a Signal? 

I. Psychophysics 

A. Definition 

B. Purpose 

II. Classical Methods 

A. Methods of limits, adjustment and constant stimuli 

B. Assumptions 

C. Advantages, disadvantages 

III. Theory of Signal Detection 

A. Response bias 

B. Decision matrix 

IV. Scaling Methods 

A. Direct 

B. Indirect 



LESSON 31 : Experimental Methods, or How Do I Know If I've 

Done It Right 

I. Eager's Experiment 

II. Research Categorization 

A. Theoretical 

B. Empirical 

1. Observation, correlation, exper imenttation 

2. Natural (field) , laboratory 

III. Variable Classifications 

A. Qualitative and quantative 

B. Independent and dependent 

C. Relevant 

1. Subject 

2. Situational 

3. Sequence 



LESSON 32 : Experimental Methods, or How Do I Control These 

Influences ? 

I. Control Techniques 

A. Subject 

B. Situational 

C. Sequence 
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II. Designs 

A. Within subject 

B. Between subjects 

1. Simple randomized 

2. Factorial 

III. Validity 

A. Internal 

B. External 



Lesson 33 : Statistics, Part I 

I. Introduction 

A. Description of total statistics 

B. Description of statistics 

C. Purpose of statistics 

II. Frequency Distributions 

A. Ungrouped data 

B. Internal data 

C. Meaning of any one score 

D. Percentile ranks 

III. Measures of Central Tendency 

A. Mean 

B. Normal curve 

C. Median 

IV. Measures of Despersion 

A. Mean deviation 

B. Standard deviation 

C. Variance 

D. Relation of 50 and normal curve 



LESSON 54 : Statistics, Part II 

i. Correlation 

A. Degree 

B. Direction 

II. Inferential Statistics 

A. Nonparametric 

B. Parametric 

C. Appropriate statistical tests 

1. t-test 

2. F test 

3 . ANOVA 



lessons 
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III. Interpretation of Results 





A. 


Practical vs. statistical significant 




B. 


One-way ANOVA 




C. 


Two-way ANOVA 




D. 


Higher order ANOVAs 


LESSON 


35: 


Review, or How Have I Done So Far? 


LESSON 


36: 


Human Factors and the Military 


I. 


AR 


602-1 


II. 


MIL 


-STD-1472C 


III. 


MIL 


-HDBK-759 


IV. 


MIL 


-STD-1474B 


LESSON 


37: 


Human Factors Test and Evaluation, or 
HFTEMAN Cross a MOAT using a HEDGE? 


I. 


HF 


Testing 




A. 


HEL TM 29-76 




B. 


MOAT 




C. 


HFTEMAN 




D. 


HEDGE 




E. 
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HUMAN FACTORS ENGINEERING SELF-PACED COURSE 
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Section I 



Introduction to the Evaluation Plan 

101. Purpose . The purpose of this evaluation plan is to 
assess the operational effectiveness of the Human Factors 
Engineering (HFE) Self-Paced Course and its potential value 
for use in the military test and evaluation community. 

102. Course Description . This course is divided into three 
major sections, Section I (Lessons 1-5) deals with the 
human's capabilities and limitations. In this section, the 
history and continued need for human factors engineering will 
be investigated. Lessons 6-10 are concerned with the physio- 
logical capabilities and limitations of human beings. Lessons 
11-13 discuss how the proper design of controls and displays 
makes use of the information learned in the previous lessons. 
Lessons 14-19 take a look at human interactions with the 
environment, followed by a review of the first section in 
lesson 20. 

The primary focus of Section II is upon the human fit- 
ting into the system. Lessons 21-27, investigate the role of 
the human engineer in various aspects of analysis, such as 
systems analysis, cost analysis, task analysis, etc. Lessons 
28-29 are concerned with the selection and training of person- 
nel. In lessons 30-34, the student will receive a short course 
on experimental techniques and statistical concepts. Finally, 
lesson 35, will offer a review of Section II. 
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Section III is entitled "Human Factors in the Military". 

In this section, lessons 36 and 37 focus on human factors 
organizations, documentation, and future application. 

In order to give the students a practical application of 
what has been presented in the course, lessons 38 and 39 will 
ask him/her to work on a "real world" problem. Finally, in 
lesson 40, a more typical review of the entire course is 
presented. 

103. Background . The HFE Self-Paced Course was developed to 
satisfy a need for increased awareness and more indepth under- 
standing of Human Factors. 

Reference (a), highlights this need and identifies a 
deficiency in the performance of various weapons systems 
"...because the DOD does not pay enough attention to logistic 
support, human factors and quality assurance during the design 
phase of the acquisition process. These problems deter the 
systems* effectiveness to defend our country in case of war." 
The Government Account Office (GAO) "therefore makes recommen- 
dations to improve the management and planning of ownership 
considerations that have an impact on the effectiveness of a 
weapon system." 

The Human Factors Test and Evaluation Manual (HFTEMAN), 
VqIs. I, II and III (reference (d)) was distributed to various 
government agencies in October 1976 by Pacific Missile Test 
Center, Pt. Mugu, California. HFTEMAN was developed to provide 
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standardization in procedures, testing and criteria in eval- 
uating human factors. It, however, assumes a basic knowledge 
of human factors for its most effective use. 

The HFE Self-Instruction course was developed in order to 
provide this basic knowledge. It has not been evaluated or 
used on a trial basis in any portion of the military prior to 
this time. 
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Section II 



Administrative Information 

201. General . General responsibilities for activities in- 
volved in this evaluation are provided in this section, as well 
as appropriate points of contact. Continuing close liaison is 
essential to timely and successful prosecution of this evalu- 
ation. 

202. Responsibilities . 

a. Naval Postgraduate School (Lt. M. M. Fleming) 

(1) Promulgate major changes to this evaluation plan. 

(2) Coordinate arrangements for HFE course subjects. 

(3) Coordinate distribution of required course 
materials to participants. 

(4) Conduct briefings for all participating if so 
requested and funded by the requesting activity. 

(5) Provide certificates of completion or equivalent. 

(6) Analyze evaluation results and make them avail- 
able to the appropriate units upon request. 

b. C0M0PTEVF0R , DEPC0M0PTEVF0R , AFT&E Center/TELH, TESTG/ 

ENAH 

(1) Furnish names and/or Student Information Form 
(SIF) numbers of participating students. 

(2) Provide point of contact for test subjects with- 
in each command (Command Course Supervisor - CCS). 

(3) Keep Lt. M. Fleming advised of students progress. 
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c. U. S. Army Human Engineering Laboratory, Pacific 
Missile Test Center (PMTC)i 

(1) Provide required number of copies of all course 
materials for distribution to subject students. 

(2) Provide point of contact to Lt. Fleming. 

(3) Provide required plans, schedules and procedural 
guidelines as indicated in Section IV. 



203. Points of Contact . 

a. Naval Postgraduate School 

Lt. Martha M. Fleming 
Evaluation Director 
Naval Postgraduate School 
SMC # 13^0 

Monterey, California 93940 
Autovon: 878-2536 

Telephone* 408-646-2536 



Cdr. Bill Moroney/ Dr. R. A. McGonigal 

Thesis Advisor 

Naval Postgraduate School 

Code 55 MP 

Monterey, California 93940 
Autovon* 878-2620/2594 
Telephone* 408-646-2620/2594 



Professor Richard S. Elster 
Secondary Thesis Advisor 
Naval Postgraduate School 
Code 54Ea 

Monterey, California 93940 
Autovon* 878-2792 
Telephone* 408-646-2792 



b. COMOPTEVFOR 

Lt. Vickie Bonnano 
Command Course Supervisor 
COMOPTEVFOR, Code 02A 
Naval Station 
Norfolk, Virginia 23511 
Autovon* 690-5598 
Telephone* 804-444-5598 
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C. DEPCOMOPTEVFORPAC 



Capt. Robert E. Sheridan 

Command Course Supervisor 

DEPCOMOPTEVFORPAC 

NAS North Island 

San Diego, California 92135 

Autovon: 951-6970 

Commercial* 

d. AFT&E Center/TELH 

LCOL P. A. Crowley 

Command Course Supervisor 

AFT&E Center/TELH 

Kirkland AFB, New Mexico 87117 

Autovon: 244-9606 

Commercial: 

e. TESTG/ENAH 

Cyrus T. Crites 
Command Course Supervisor 
2620 TESTG/ENAH Stop 239 
Edwards AFB, California 
Autovon: 350-333^ 

Commercial: 

f. Pacific Missile Test Center, U. S. Army Human 
Engineering Laboratory 

Cdr. Tom Jones 

Pacific Missile Test Center 

Code 1226 

Pt. Mugu, California 93042 
Autovon: 351-8981 

Commercial: 805-982-8981 

g. ASD/ENECH 

Dr. Richard Shiffler 
ASD/ENECH 

Wright Patterson AFB, Ohio 45433 

Autovon: 785--6010 

Commercial: 



h. Capt. Don Loose 

Electronics Systems Division 
Hanscom AFB, Massachusetts 01731 
Autovon: 478-2825 

Commercial: 
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Section III 



Scope of the Evaluation 

301. Objectives . In accordance with the recommendations of 
reference (a), standardized objectives upon which this course 
is to be evaluated will follow the procedures found in refer- 
ences (c) through (j) and as follows* 

a. Course Objectives . The course objectives must be 
developed according to accepted standardized structure in order 
to prepare the student to perform his/her job in testing and 
evaluating human factors. 

b. Course Material Design. The format of the course 
materials must aide the student in accomplishment of the stated 
learning objectives. The following areas are included in this 
evaluation* 

1. Sequence and structure 

2. Figures and Tables 

3 . Symbology and legends 

4. Layout and format 

c. Course Content . The course content must support the 
course learning objectives. 

d. Course Presentation. The design of the course pre- 
sentation aids the student in accomplishment of the course 
learning objectives without the aid of an on-site instructor. 

In addition to the above, future plans for the following 
will be reported* 

a. Course Availablilitv . The course materials should be 
maintained in sufficient quantity to meet the needs of those it 
will serve. 
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b. Course Maintainability . The course materials must be 



in accordance with the most recent version of the references 
upon which it is based and human factors design developments. 

c. Course Supportabilitv . Assistance must be available 
by correspondence with the issuing agency. Feedback must be 
provided to the student on his/her performance. The life cycle 
cost of the course must be available for future planning and 
assessment. 

302. Criteria . The criteria below are in accordance with ref- 
erences (e) through (i) and other resource materials listed in 
the bibliography. 

a. Clours e Objectives . The course learning objectives 
should have all the characteristics listed below. 

1. Objectives must be a statement of student behavior 
(action), such as the creation of a product or some other overt 
act, which can be accepted as evidence that the intended out- 
come has occurred. 

2. The behavior must describe specifically all out- 
comes that will demonstrate that learning has occurred. 

3. The student behavior called for must be capable 
of observation and evaluation within the learning and testing 
environments. 

4. The objective must be stated in learner rather than 
teacher terms, i.e., actions which the student will perform 
rather than what the instructional materials will "say or do". 

5. There must be a standard against which the student 
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behavior will be measured. It must be fully specified. 

6. The statement of the conditions under which the 
student behavior will occur must be fully specified. 

Additionally, students must be able to satisfy the ob- 
jectives during the normal performance of their jobs in human 
factors test and evaluation. 

b. Course Material Design . 

1. Sequence and structure . The learning objectives 
should be arranged in the sequence in which instruction will be 
presented to the student. 

2. Figures and Tables . Sufficient information should be 
provided with each figure and table to allow the student to 
apply it as directed within the sourse materials and during 
performance of his/her job when left only with the student 
supplement. 

3. Symbology and Legends . The symbols and legends 
presented in the course materials should be implicit 

and standardized where possible. 

k . Layout and format . The layout and format must aide 
the student in accomplishment of course objectives and encourage 
motivation and desire to complete the course. 

c. Course Content . The mean score of the course groups' 
second diagnostic tests must be at least one standard deviation 
above the mean score of the control groups' second diagnostic test. 

d. Course Presentation . The instructions and verbal context 
of the course materials must allow accomplishment of course 
objectives and course completion without the aide of sin on-site 
instructor. 
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303. Evaluation . 



a. Site selection . The evaluation of the HFE course will 
be conducted at various evaluation sites which are involved in 
test and evaluation of systems involved in the acquisition 
cycle. Testing at these sites will provide a realistic environ- 
ment in which to exercise the course. The course is designed 
for personnel with this same job type and responsibilities. 

b. Personnel selection . Subjects participating in this 
evaluation will be drawn from three military commands whose 
primary mission is operational test and evaluation. Some per- 
sonnel will be volunteers, while others may be assigned to 
participate. 

1. Specialty area . It is anticipated that the 
evaluation subjects will be drawn from the various divisions 
within each command (air, surface, subsurface and special war- 
fare) . 

2. Special requirements . No constraints regarding 
rate, rank, AFS, grade, educational level or prior experience 
will be placed on participants. The only requirement is that 
each subject be actively involved in planning and performing 
operational test and evaluation. 

c. Data Collection. Data sheets, course diagnostic 
tests and questionnaires for use in this evaluation are con- 
tained in Annex A. Copies will be distributed to Command 
Course Supervisors prior to beginning the evaluation. More 
specifics regarding the actual evaluation procedures are pro- 
vided in Section IV. 
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304. Limitations to Scope . While the HFE Self-Instruction Course 
consists of 40 lessons, only the first 20 are scheduled to be 
evaluated. All activities taking part will be provided with 
all 40 lessons in order to provide subjects the opportunity to 
complete the course at a pace faster than that recommended by 
Pacific Missile Test Center. 

The criteria established for assessing the adequacy of the course 
objectives require a prolonged period for a full and complete 
evaluation. This evaluation lasts only for a period of 30 
days and will only analyze the completed questionnaires of the 
participating subjects. This will only provide a preliminary and 
limited assessment. Therefore, it is recommended that a second 
questionnaire be drafted and distributed to the course par- 
ticipants six months after course completion. The subjects then 
will have had an opportunity to apply what the course taught 
them sind make a more accurate assessment of of its v al ue . 
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Section IV 



Evaluation Guideline 

401. Evaluation participants and HFE Course Materials . The 
HFE course is designed to provide Human Factors training to 
personnel responsible for planning and conducting testing in 
human factors. This is part of the job of an Operational Test 
Director (OTD) as stated in references (c) and (j). The 
personnel aiding in the evaluation of this course should therefore 
have the same responsibilities. This is the reason for the 
selection of the particular evaluation sites. This course 

was designed for the job and is not limited as to rate, rank. 

Air Force Specialty (AFS), grade» educational level or prior 
experience of the participant. 

402. Evaluation Procedures . The HFE course will be conducted 
as follows: 

a. Student Information Forms (SIF's) must be filled out 
by the CCS for each course participant (See Annex A). At that 
time, each student will be assigned a 4-digit code. The first 
two digits (from the left) identify each individual command: 
CQMOPTEVFOR OlXX 
DEPCOMOPTEVFOR 02XX 
AFT&E Center 03XX 
TESTG/ENAH 04XX 

The last two digits are then assigned in numerical sequence to 
each participating student: 

Student 1 XX01 

Student 2 XX02 
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Example* COMOPTEVFOR — 3 students 



Student 1 



0101 



Student 2 



0102 



Student 3 



0103 



This HFE SIF course number should be retained by both student 
and supervisor. It should appear on all correspondence, tests, 
questionnaires and SIF's of each student. 

b. After an SIF has been completed and prior to beginning 
the course, each student will be given a diagnostic test, to 
determine the individual level of each student before beginning 
the course. Each CCS will administer the test and make the pur- 
pose of the test clear to each participant. 

In addition, the student must be instructed not to guess if an 
answer to a particular question is not known — the answer 
should be left blank. The tests will be mailed back to NPS 
immediately upon completion for retention and evaluation. 

c. Following the diagnostic test, each student will be 
provided with the course materials and references. A schedule 
for timely completion of the course will also be provided. Upon 
completion of the course, a final diagnostic test will be ad- 
ministered to each student, by the CCS. This test is also 
designed around the course objectives. While the questions are 
not the same as the first diagnostic test, they will basically 
cover the same material. This test is to be mailed back to NPS for 
scoring. The performance of the student on each of the diagnos- 
tic tests will then be compared. Should the participant wish to 
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to know his/her score on the test(s), it/they will be provided 
by the Evaluation Director at NPS upon request. 

d. A questionnaire will be provided to each student who 
begins the course to provide feedback on individual opinions 
about the course after each has completed it. 

e. A questionnaire will also be provided to each CCS. 

f. Questions asked by the student should be recorded 
by the CCS in the provided log. Each question should also be 
identified with an SIF number. 
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403. Test 01 . Course Objectives . 



a. Ob ,i ec t . To assess the extent to which the course 
objectives support the tasks required of a human factors test 
and evaluation agent. 

b. Procedure . A task analysis of the human factors 
specialist will be compared to the course objectives. Ques- 
tionnaires regarding the course objectives will be distribu- 
ted to various resources dealing with human factors test and 
evaluation and curriculum design. 

c. Data Requirements . A task analysis of a human 
factors test and evaluation specialist and completed question- 
naires are required. 

d. Data Analysis . Course objectives will be compared 
to the task analysis to evaluate how adequately the course 
objectives support the person responsible for performing 
human factors test and evaluation. 

404. Test 02 . Course Materials Format . 

a. Object . To assess readibility, standardization and 
clarity of format according to references (e) through (i). 

b. Procedure . Questions directed to the CCS by each 
participant relative to the HFE course for clarification or 
explanation will be recorded. Questionnaires will be provi- 
ded to students and CCSs upon course completion and/or end of 
test period, which ever applies. 

c. Data Requirements . All recorded student questions, 
questionnaires and comment sheets will be forwarded to the 
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to the Evaluation Director at NPS, upon completion or end of 
test period which ever is applicable. 

d. Data Analysis . Questionnaire responses will be 
evaluated and the results tabulated. (See Annex B for further 
details. ) 

405. Test 03 . Course Content . 

a. Object. To assess the extent to which the course 
content supports the course objectives. 

b. Procedure . Two diagnostic tests will be administer- 
ed to each student; one prior to beginning the course and one 
following its completion. A questionnaire will also be provi- 
ded at course completion. The SIP number will be placed on 
each. 

c. Data Requirements . Participant test answer sheets 
and completed questionnaires must be returned to the evalua- 
tion director at NPS as they are turned in to each supervisor. 

d. Data Analysis . A mean and its standard deviation 
will be computed for each of the diagnostic test scores. The 
mean scores will then be compared. The individual responses 
to test questions will also be compared. 

Both diagnostic tests will also be administered to 
a control group. These individuals will not be talcing the 
course. The purpose will be to substantiate the reliability 
of the tests, and to assess whether the scores of the course 
participants change more than do scores of the control group. 



154 



IV- 5 



Individual questionnaires will also be evaluated and 
analyzed. (See Annex B for further details). 

406. Test 04 . Course Presentation . 

a. Object . To assess the clarity of the presentation 
of course content, in order to allow the student to complete 
the course without the aid of an on-site instructor. 

b. Procedure. See Test 02. 

c. Data Requirements . See Test 02. 

d. Data Analysis . See Test 02. 

407. Test OS . Course Availability . 

a. Object . To report on the plans for making course 
materials available to future students. (Implementation and 
Control) . 

b. Procedure ♦ Review the plans PMTC would recommend 
for making course materials available for general use. 

c. Data Requirements . Provision of the implementation 
and controls plans from the issuing agency will be required. 

d. Data Analysis . The response of the issuing agency 
will be reported. No other analysis is intended. 

408. Test 06 . Course Maintainability . 

a. Object . To report on the plans for keeping the 
instruction manuals up to date with the applicable military 
standards it references and latest human factors developments 
and procedures. 

b. Procedure . Review the plans of the issuing agency 
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for scheduling updates in order to keep the course content 
current. 

c. Data Requirements . Provision of the course main- 
tenance plans will be required of the issuing agency. 

d. Data Analysis . The response of the issuing agency 
will be reported. No other analysis is intended. 

409. Test 07 . Course Sup-portability . 

a. Object . To assess the plans and procedures for 
providing adequate support to students taking the HPE Self- 
Paced Course. 

b. Procedure . Review the plans and procedures provid- 
ed by the issuing agency for supplying the support required 
by the student for successful completion of the course. 

c . Data Requirements . The issuing agency will pro- 
vide the required documents to the NPS Evaluation Director 
for review. 

d. Data Analysis . See Test 06 . 
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Section 5 
Report ( s ) 



501. General. The results of this HFE Self-Paced Course 
Evaluation will be provided in the Evaluation Director's 
thesis. 
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ANNEX 3 



CCS LOG FCRfcAT 
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INTRODUCTION 



FOR 

COMMAND COURSE SUPERVISOR (CCS ) 

You have been identified by your command to be my point 
of contact with your command in the evaluation of the Human 
Factors Engineering Self-Instruction Course during the period 
1 September - 31 October 1981. The actual length of the 
evaluation at your command may vary depending on how quickly 
the individual student participants complete the course. 

In reading the CCS Guidelines, you will find that your 
responsibilities, very basically, consist ofi 

(1) Filling our Student Information Forms (SIF) 

a. Assigning SIF identification numbers. 

(2) Distributing and collecting diagnostic tests/ 
questionnaires, comments and course materials. 

(3) Maintaining your CCS log 

a. Student Progress Record 

b. Student Question and Comment Record 

c. Student Questionnaires 

d. CCS Questionnaires 

e. Evaluation Completion Checklist 

(4) Contacting me for the answers to any questions 
which the student participant may ask that are not obvious 
to you or have not been provided in the material sent 
you. My phone number is* Autovon* 878-2536 
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(5) Returning all materials and correspondence as 
each is completed; but no later than 1 November 1981 . 



Mail to t 



Lt. Martha M. Fleming 
Naval Postgraduate School 
SMC # 1340 

Monterey, Cal, 93940 
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EVALUATION GUIDELINES 



FOR THE 

COMMAND COURSE SUPERVISOR (CCS) 

1. Student Information Forms (SIF) and Diagnostic Test >yl : 

It is recommended that you arrange to meet with participants in 
a group. 

a. Before the meeting , it is essential that you« 

(1) Read the SIF Instruction sheet to be sure that you 
understand what is wanted in each question. 

(2) Fill in the SIF numbers on each form. 

b. During the meeting t 

(1) Explain what the evaluation is all about. Hand out SO-l/4. 

(2) Hand out the "Privacy Act Statement" (DA Form 4363-R, 

1 May 1975) » which is attached to the front of the SIF. This 
form basically promises them confidentiality of the information 
they provide in the SIF and the questionnaires. As long as they 
follow directions provided, only the participant and the Eval- 
uation Director at the Naval Postgraduate School will see their 
responses to questionnaires and test results. As it states at 
the bottom of the form, they may keep DA Form 4368 -R, but do 
need to return the attached SIF. 

(3) SIFj Point out that "name" is optional. The rest 
should be self-explanatory. If not, you have the SIF Instruction 
Sheet which should provide you with any necessary clarifying 
information. 
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(4) Diagnostic Test Number 1» After the SIF has 
been collected! 

(a) Pass out the "Diagnostic Test Number 1". 

(b) Read through the instructions with the 
students orally. Be sure to highlight those statements or 
words which are underlined. 

(c) Offer them the opportunity to take the 
test in a room where it is nice and quiet? where they won’t 
be disturbed. They may take the test back to their desks or 
spaces, which ever they would feel most comfortable doing. 

Even though the test is designed to see how much they know 
about the world of human factors before taking the course, 
the test is. not going to be graded, per se, as there is no 
standard against which to measure. The students are ON THEIR 
HONOR to do the test by themselves without reference material 
or other help. The test must be turned in by the end of the 
working day. 

(d) Should some choose to take the test then 

and there, position yourself in the room so that you are avail- 
able to answer and record questions the participants ask of 
you. Do NOT walk up and down the aisles. This is in an 
effort to relax the tensions many people have when they sit 
down to take any kind of a test. 

(e) Remember, there is no time limit , so 
allow yourself adequate time for proctoring the test. For 
planning purposes, the test is designed so that even with no 
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previous background in human factors, the test should be 
easily completed within 45 minutes. Students, however, are 
not required to complete the test within this time period. 

2. Command Course Supervisor Log : The CCS Log consists of 

the following i 

SIFs for all participants 
Student Progress Records 
Student/CCS Questions and Comments Record 
Unanswered Student Questionnaires 
CCS Questionnaire 
Evaluation Completion Checklist 
FOR FURTHER CLARIFICATION t 

a. SIF i Keep these in the appropraiate section of your 
log. Dividers have been provided. See also the ”SIF Instruc- 
tion” sheet. 

b. Student Progress Records 1 Record the completion date 

fori 

(1) SIFs 

(2) Diagnostic (D-Test) Test #1 

(3) Each lesson book 

(4) Diagnostic Test #2 

(5) Student Questionnaire 

(6) CCS Questionnaire 

Be sure to keep track of the proper SIF number for each student. 

c . Student Questions and Comments Record i Each time 
either you or a student have a question or comment, record 
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the following i 



(1) date. 

(2) SIP number. 

(3) the question or comment. 

(4) the answer provided. 

d. Student Questionnaires ! Blank questionnaires have 
been provided so that they are available as each student 
completes the second diagnostic test. These may be taken 
away for completion, 
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EVALUATION OUTLINE 



I. Purpose of the Evaluation : The purpose of this evaluation 

is to assess the operational effectiveness of the Human Factors 
Engineering (HFE) Self-Instruction Course and its potential 
value for use in the Fleet test and evaluation community. 

II. Course Description ! This course is divided into three 
major sections. Section I (lessons 1-5) deals with the human's 
capabilities and limitations. In this section, the history 
and continued need for human factors engineering will be in- 
vestigated. Lessons 6-10 are concerned with the physiological 
capabilities and limitations of human beings. Lessons 11-13 
discuss how the proper design of controls and displays makes 
use of the information learned in the previous lessons. Lessons 
14—19 take a look at the human being as he/she interacts with 
his/her environment, followed by a review of the first section 
in lesson 20. 

The primary focus of Section II is upon the human as he/she 
fits into the system. Lessons 21-27» investigate the role of 
the human engineer in various aspects of analysis, such as 
systems analysis, cost analysis, task analysis, etc. Lessons 28-29 
are concerned with the selection and training of personnel. In 
lessons 30-34-, the student will receive a short course on 
experimental techniques and statistical concepts. Finally, 
lesson 35» will offer a review of Section II. 

Section III is entitled "Human Factors in the Military". In 
this section, lessons 36 and 37 focus on human factors organ- 
izations, documentation, and future application. 
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In order to give the students a practical application of 
what has been presented in the course, lessons 38 and 39 will 
ask him/her to work on a ‘real world' problem. Finally, in 
lesson 40, a more typical review of the entire course is pre- 
sented. 

III. Eurpose of the HFE Course * The HFE Self-Instruction Course 
was developed to satisfy a need for increased awareness and more 
indepth understanding of Human Factors. 

The Comptroller General, in his report to the Congress of 
the U. S. , dated January 29, 1981, (PSAD-81-17) , entitled 
Effectivenss of U. S. Forces Can Be Increased Through Improved 
Weapon System Design , highlights this need and identifies a 
deficiency in the performance of various weapons systems "be- 
cause the Department of Defense does not pay enough attention 
to logistic support, human factors and quality assurance during 
the design phase of the acquisition process. These problems 
deter the system's effectivenss to defend our country in case 
of war. 

GAO therefore makes recommendations to improve the manage- 
ment and planning of ownership considerations that have an 
impact on the effectiveness of a weapon system." 

The HFE Self-Instruction course was developed in order to 
provide a basic knowledge of human factors and how to test for 
its effectiveness. The course has not been evaluated or used 
on a trial basis in any portion of the Navy prior to this time. 
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IV. Summary of Evaluation Steps ; It is anticipated that each 
of the 40 lessons will require on hour completion time. While 
the course developers recommends 40 working days to complete the 
entire course, time constraints levied necessitate completion 
prior to 31 October 1981. 

(1) Introduction by Command Course Supervisor (CCS) 

(2) Receive Privacy Act Statement 

( 3 ) Fill out Student Information Form (3IF) 

(4) Receive instructions for and take Diagnostic Test #1 

a. student supplement 

b. lesson booklet 1-5 

c. applicable references 

(6) Return each lesson booklet as it is completed and pick 
up the next one in the series, until all 40 lessons are completed. 

(7) Receive and take diagnostic test number 2 

a. return test in sealed envelope 

(8) Fill out student questionnaire 

a. Indicate whether or not test results are desired 

b. Be sure SIF number is visible on envelope 

c. Return questionnaire to CCS in sealed envelope 
within two working days or not later than 31 October 1981, which 
is soonest. 

Should you experience any repeated difficulty or delays in 
locating your CCS, feel free to contact me. I do not have my own 
phone, however, a secretary will take the message and I will 
get back to you. Please remember that there is a three hour 
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time difference between the East and West Coasts, 
our 1645 and the secretaries leave at I 63 O. 



Your 13^5 is 



Lt. Martha M. Fleming 
Naval Postgraduate School 
SMC #1340 

Monterey, California 93940 

Autovon: 878-2536 

Commercial: 408-646-2536 
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DIAGNOSTIC TESTS INSTRUCTIONS 



Purpose i The purpose of the first diagnostic test is simply to 
see how much you know about human factors before you begin the 
Human Factors Engineering (HFE) Self-Instruction Course. This 
is why we call it a diagnostic test. After you have completed 
the HFE course, a second diagnostic test will be given. The 
purpose of this test is to see how successfully the HFE course 
has increased your awareness of the various facets of human 
factors. 

Guidelines i In order to perform an accurate comparison analysis of 
the two tests, there are several things which we must ask of you 
while taking these diagnostic tests i 

(1) Answer the questions to the best of your knowledge, 
do not get anyone to help you or explain the question to you. On 
the first diagnostic test, you are not expected to know the 
answers. We hope that the HFE course will help you answer the 
questions on the second test. 

(2) Do not guess ! Please, if you understand the question, 
but do not know the answer, select the option which indicates 

that you don't know. We realize that this is against your human 
nature and therefore is difficult to do, but we ask that you try. 

(3) This is an evaluation of the HFE Course , not of the 
course participants. You do not flunk or pass; the course does. 

If it is so indicated by your tests, questions, comments and 
questionnaires, the course will return to Pacific Missile Test 
Center for rework. Therefore everything you have to say about 
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the course will help us make sure it will be a good one when it 
"hits the streets". 

(4) If a question does not make sense and needs clarification, 
see your Command Course Supervisor (CCS). 

(5) Your CCS will be recording your questions as you 
ask them. If you have a question, then obviously the materials 
didn't make it clear enough and need to be changed. You can help 
your CCS by writing your question down -- don't forget to put 
your SIF identification number on it. 

(6) Since the HFE course is designed to fit the needs 
of the job of testing and evaluating human factors, rate, rank, 
grade or educational level should not make a difference. Again, 
if it does, the course materials need to be improved . 

(7) Where you take the test is up to you. Your CCS will 
provide you time and a place to take the test, giving you easy 
access to him/her for questions and a place where you will not 
be disturbed. You may elect to take the test back to your desk 
or office, however, remember not only the HONOR SYSTEM of para- 
graph number one, but also that the test must be returned to 
the CCS in a sealed envelope by the end of that working day. 

(8) Should circumstances beyond your control prevent you 
from finishing the course materials, you still must ? 

a. Notify your CCS 

b. Take diagnostic test #2 

c. Fill out the student questionnaire 

Since each test is progressively designed you will still provide 
valuable data by doing (b) and (c) above. 
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(9) Do you want to know your test results? You may 
make that choice while answering the student questionnaire. Your 
CCS will not have the results as the tests are analyzed by the 
Evaluation Director at the Naval Postgraduate School. They 
will be forwarded to you at the end of the evaluation upon 
receipt of your request. 

(10) Do you have any questions? If so address them to 
your CCS now. 

In Summary : 

(1) Do the best you can . 

(2) Do not guess. 

(3) The course is evaluated, not the participant. 

(4) Questions and comments will be recorded in 
logs maintained by your command's Course Supervisor. 

(5) Rate, rank, grade, AFS, educational level and 
prior experience should not make a difference. 

(6) Take your Diagnostic Test #1 where you want, but 
turn it in by the end of that working day. 

(7) Once you start the course, plan on taking diagnostic 
test #2 and filling out the student questionnaire. Return the 
test by the end of that working day and the questionnaire within 
two working days. 

(8) Want your test results? Tell us so in the questionnaire. 

(9) Questions? Ask the CCS. 

•♦THANK YOU for participating in this evaluation; without you 
it wouldn't be happening. 
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COMMAND COURSE SUPERVISOR (CCS) 
EVALUATION COMPLETION 
CHECKLIST 



1. Administrative Wrap-Up 

a. Be sure that each of the following has been completedi 

(1) SIF identification numbers on all completed items. 

(2) SIF for each participant. 

(3) Student Progress Records. 

(4) Student/CCS Question & Comment Record. 

(5) Diagnostic Tests #1 and #2 from EACH PARTICIPANT 
beginning the Human Factors Engineering Self-Instruction Course . 

(6) Student Questionnaire from EACH PARTICIPANT begin- 
ning the course. 

b. Retain some way to identify students with their SIF 
identification numbers. This will be necessary should any partici- 
pants request their test scores or evaluation results. We do 

not wish to know them, only that you be able to pass on the 
information upon its receipt. 

2. Return the following together in one package by AIR MAIL — 
FIRST CLASS — NO LATER THAN 1 NOVEMBER 1981 : 

a. CCS Logs and all completed copies ofi 

(1) Student Information Forms (SIF) 

(2) Diagnostic Test #1 

( 3 ) Student Progress Record 

(4) Student/CCS Question and Comment Record 
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(5) Diagnostic Test #2 

(6) Student Questionnaires 

(7) CCS Questionnaire 

3. Return the following together in a separate package by 
PARCEL POST — Bookratei 

a. All course materials and extra , uncompleted forms and 
questionnaires i 

(1) Student Supplement 

(2) Booklets for lessons 1-40 

(3) Course references (MIL-STDs and TECHMEMOs) 

(4) SIFs 

(5) Diagnostic Tests 1 and 2 

(6) Student Progress Records 

(7) Student/CCS Question and Comment Records 

(8) Student Questionnaires 

(9) CCS Questionnaires 
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STUDENT PROGRESS RECORD 



Completion Date (month & day) 
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STUDENT QUESTION & COMMENT RECORD 



DATE 


SIF # 


Student's Question/Comment 
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DIAGNOSTIC PRETEST 
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DIAGNOSTIC PRETEST INSTRUCTIONS 



PURPOSE . The purpose of this test is merely to identify the level of 
your familiarity with human factors engineering BEFORE you take the 
self-paced course. A second diagnostic test will he given after les- 
son 20. If the course fulfills its purpose, the level of your human 
factors engineering familiarity should increase. The results of the 
second test should support this theory. 

FORMAT . This prediagnostic test has eight pages and 28 questions. If 
the test is designed correctly, it should require less than one hour to 
complete. 

DESIGN . There are several different types of questions: 

1. Multiple Choice - These questions are identified by such terms 
as "CHOOSE" or "SELECT". One or more answers may be correct. Identify 
your choice by placing an "X" in the space provided. 

2. Matching The word "MATCH" denotes such questions. Only one 
answer should be chosen for each term on the left. Some of the phrases 

or acronyms on the right may be left unused; don't worry, the question was 
designed that way. 

3. Fill in the Blank - Words such as "NAME", "IS CALLED" or "LABEL" 
identify these types of questions. Sentences requiring completion are also 
examples of "fill ins". You should not need to use more than 5 words to 
answer those questions. 

Short Answer - These questions are identified by such terms as 
"EXPLAIN", "DESCRIBE", "DEFINE", "WHY", "WHAT, and/or ask for examples. 
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Be brief and to the point, but remember, that someone else must be able to 
understand your answer; so work on its clarity. 

5. Application - One question asks you to redesign, if necessary, 
a display mechanism. Should you decide it is needed, DRAW the redesigned 
mechanism in the space provided. Remember, however, you may be happy with 
the displays as they are presented. Should that be the case, do nothing. 

TEST VALIDITY . It is essential that you do this test by yourself, without 
the use of references or outside resources. To do so will affect the 
validity of the analysis method selected for this evaluation. 

GUESSING . If you want to guess, go ahead. You would anyway, even if I 
told you not to do so. 

GRADING . Keep in mind that the test is being graded, not you. Should 
you like to know the score the test made, you will have the opportunity 
to let us know at the end of the evaluation period. 
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MULTIPLE CHOICE: Choose the best answer(s). One or more than one answer 

may be correct. Place an f X f by your choice(s). 

A. The overall objective in taking a human factors engineering course is: 

1. to obtain the human factors engineering background 

necessary to do your job well. 

2. to understand a person’s specific capabilities and 

limitations. 

3. to understand the precise military procedures involved 

in a human factors program. 

4. to understand the importance of applying the concepts and 

priorities of human factors engineering in your job. 



S. The main reason for poor system performance is human error which is 
caused by: 

1. inadequate consideration of human performance capabilities, 

skill limitations and response tendencies. 

2. little standardization of ’controls 1 , (knobs, levers, etc.). 

3. the fast developing pace of new technologies with which 

the human’s capabilities cannot keep up. 

4. people with too little education and too few skills being 

brought into the armed services. 

5. inadequate human factors input into system man-machine 

interface. 



C. There are three main areas of the human’s information processing capabilities. 
MATCH the terms on the left with the appropriate example (s) on the right. 



1. selective attention 

2. physiological processing 
limitations 

3. channel capacities 



a. hear a signal above 20,000 CPS 

b. interrupting normal procedures 
to react to an alarm bell. 

c. listening to four incoming 
ship to ship and two secure 
voice radio messages simultan- 
eously on the bridge of a ship 

d. breaking out in a cold sweat 
when required to perform a 
specific duty. 



D. MATCH the sensory modalities with the related stimulus dimensions: 



1. vision 


i. 


saltiness 


2. audition 


2. 


hue 


3- odor 


3. 


pure tones 


4- taste 


4. 


smell 



E. EXPLAIN WHY each of the following areas is important in human factors 
design evaluation: 

1. information processing capabilities: 



2. sensory modalities: 



F. The science dealing with measurement of the physical features and functions of 

the body is called . 

G. The term which refers to the measurement of human body dimensions in a fixed 

position is . 

H. NAME the term which indicates that body dimensions are determined from 

body positions which occur with movement- 

I. EXPLAIN WHY body dimensions are an important human factors consideration. 
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J. LIST TWO examples which support your definition on the importance of 
body dimensions as a human factors consideration: 

1 . 



2 . 



K. NAME five environmental considerations which may affect a human’s performance: 

1 . 

2 . 

3. 

4. 

5. 

L. CHOOSE which phrase deals with the principle of anthropometric design: 

1. use of population stereotype 

2. sequence of use analysis 

3. 5th - 95th percentile humans 

4. design for the extreme individual 

M. CHOOSE which phrase deals with the principle concerned with the general 

design layout. 

1. sequence of use 

2 . length of use 

3. vigilance capabilities of the operator 

4. frequency of use 

5. function of a component 



N. NAME FOUR methods of gathering human factors data on human activities: 



1 . 

2 . 

3. 



4. 



0. What is the purpose of link analysis? GIVE AN EXAMPLE. 



P. Knobs should have basic standardized functions. MATCH the terms on the 
right with the figures on the left. 



Column 
( 1 ) ( 2 ) 



1 . 



Column (1): 

a. fractional reaction 

b. discrete reaction 

c. multiple reaction 




2 . 




Column (2) : 

* d* * less than 1 full turn and 

position is not very important 

e. less than 1 full turn where 
position is important 

f. at least 1 full turn 
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Q. CHOOSE which of the combined effects of temperature and humidity 
the human body will adapt to most quickly: 

1. extreme heat and high humidity 

2. extreme cold and high humidity 

3. extreme heat and low humidity 

4. extreme cold and low humidity 



R. LABEL the following display mechanisms as: 
(1) quantitative OR qualitative AND 




a. aircraft 'turn and 
bank indicator' 



b. automobile 
temperature 
control 



c. submarine depth 
gauge 



( 1 ) 



( 1 ) 



( 1 ) 



( 2 ) 



( 2 ) 



( 2 ) 



S. If you would improve the designs presented in the figures above, SHOW what 
the recommended design change would look like in the space provided below: 
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T. DEFINE the term noise: 



U. There are eight concepts for evaluating control design. Control coding 
is one of these. NAME FOUR of the six factors which are used when 
evaluating control coding: 



1 . 

2 . 

3. 

4 . 

V. Another of the eight concepts for control design deals with direction of 
control movement. Briefly EXPLAIN WHY this is important: 



W. When do the greatest performance decrements occur during a watch-standing 
duty? CHOOSE the best answer(s): 

1. first 30 - 60 minutes of the watch 

2. last 30 - 60 minutes of the watch 

3. last 30 - 60 minutes before lunch break 

4. workspace temperature is between 75°F - 82°F 

5. several short rest periods are given between watches 



X. What is a task analysis? EXPLAIN: 
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Y. Why is a task analysis so very important in the field of human factors 
design and evaluation? 



Z. MATCH the acronyms with the associated terms: 



1. radiation 


a. PPM 


2. frequency 


b. PSI 


3. atmospheric contamination 


c. RAD 


4. pressure 


d. CPS/Hz 


5. intensity 


e. CLO 


6. hearing loss 


f. dB/PNDB 


7. insulation 


g. PTS/TTS 



AA. NAME TWO key military standards (MIL STDS), handbooks or U. S. Army 

Human Engineering Laboratory Technical Memorandums which provide guidelines 
and/or criteria for: 

1. control design 

2. environmental factors/ 

considerations 

3. anthropometric data 



BB. NAME ONE key military standard (MIL STD), handbook or U. S. Army Human 

Engineering Laboratory Technical Memorandum which provides guidelines and/ 
or criteria for: 

1. display design __ 

2. aural non-detectability 

3. noise limit selection 



4. control color coding 



K 
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5 . vigilance 

6. criticism regarding 
stated criteria gov- 
erning human exposure 
to carbon monoxide (CO) 



You have now completed your first diagnostic test. Don't feel badly if you 
think that you didn't know many of tne answers. You weren't expected to, remem- 
ber? This test has been a brief introduction as to what you can expect to 
learn as you proceed through the HUMAN FACTORS ENGINEERING SELF-INSTRUCTION 
COURSE. The first lesson booklet and the student supplement will be handed 
to you when you turn this test in to your COMMAND COURSE SUPERVISOR. 

GOOD LUCK !!!!!! Hope you enjoy the course and THANK YOU for your partici- 
pation and support. 
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DIAGNOSTIC POST TEST 
INSTRUCTIONS 



PURPOSE . The purpose of this test is to measure your familiarity 
with human factors. Some of you will be taking this AFTER com- 
pleting the Human Factors Engineering Self-Paced Course 
Others of you in the "control group" will be taking this test 
without taking the course. Those of you in the "control group" 
will be providing a baseline from which to measure the "course 
group's" test results. 

FORMAT. This diagnostic post test has 7 pages and 29 questions. 

If the test is designed correctly, it should require approximately 
one hour to complete. 



DESIGN . Eventually this course will prepare you to develop a gen- 
eral human factors evaluation outline. This requires a thorough 
familiarity with the following areas of human factors design and 
evaluation: 



Test objectives 
Criteria 

Analytical methods 
Resource documents 



Terms 

Definitions 
Design principles and 
concepts 



The content of this test will establish your readiness for course 
completion. 



There are several different types of questions: 

1. Multiple choice . These questions are identified by the 
term "CHOOSE". None, one or more answers may be correct. Identify 
your choice by placing an 'X' in the space provided. 
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2. Fill- in- th e-blank . Words such as "NAME" , "LABEL" or "LIST 



identify these types of questions. You should nox need to use 
more than five words to answer these questions. 

3 . Short answer . The questions are identified by such terms 
as "EXPLAIN", "DESCRIBE", "DEFINE", "'WHY", "WHAT" and/or ask for 
examples. Be brief and to the point, but remember, that someone 
else must be able to understand your answer; so work on its clarity. 

i+. Application . One question asks you to redesign, if necess- 
ary, a display mechanism. Should you decide it is needed, DRAW 
the redesigned mechanism in the space provided. Remember, how- 
ever, you may be happy with the display as it is presented. 

Should that be the case, do nothing. 

TEST VALIDITY . It is essential that you do this test by yourself 
without the use of references or outside resources. To do other- 
wise will destroy the validity of the analysis method selected 
for this evaluation. 

GUESSING . If you want to guess, go ahead. You would anyway, 
even if I told you not to do so. 

GRADING . Should you like to know the results of the test, you 
will have the opportunity to let me know at the end of the evalu- 
ation period. 
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A. The first item of interest is the TASK ANALYSIS. 

1 . When should a task analysis be done? 



2. 3y whom should the task analysis be done?, 



3. Describe what information the task analysis provides. 



3 . SELECTION 0 F PERSONNEL for participation in each test system 
evaluation is an area of critical importance. 

1 . Explain why you agree or disagree with this statement. 



2. Name two resource documents that will aid you in selection 

of test subjects: a. b. 

3. Briefly describe the basic characteristics of a typical 

test subject; 



C. WORKSPACE DESIGN AND ARRANGEMENT is the third area of concern. 
1. Name two resource documents that will aid you in evalu- 
ating the physical layout of a workspace (i.e. cockpit or 
bridge). a. b. 
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2. Name the analysis technique recommended for evaluating 
the arrangement of components within a workspace. 



3. Name the aspect of evaluating workspace design which con- 

siders the physical characteristics of intended human 
operators. 

4. Name the two human body dimensions measured when evalu- 
ating the workspace design and arrangement. 

a. 

b. 

D. Evaluating CONTROL PANEL DESIGN should be next on your list. 

CONTROLS, DISPLAYS, and AUDIO/VISUAL ALARMS are items which 

require specific attention. 

1. Name two resource documents that will provide valuable 
guidelines for evaluating a control panel. 

a. 

b. 

2. Name the analysis method/technique which is recommended 

for determining whether the controls and displays are 
positioned optimally: 

3 . List the guidelines recommended when deciding whether to 
use an audio alarm or a visual alarm. (DO NOT name re- 
source documents — list the actual guidelines. ) 

a. 

b. 



c. 



4. List six of the eight CONCEPTS for evaluating CONTROL DESIGN: 



a. 



b. 



c. 



d. 



e. 



f. 



5- Select one of the concepts from question 4 and name four 
factors considered when evaluating this concept. 

concept * a. 

factors » ( 1 ) 

( 2 ) 

( 3 ) 

( 4 ) 

6 . Select another one of the concepts listed in question k 

and explain how this concept could be critical in preventing 
human errors. 

concept : a. 

explanation : 



7. There are three classes of knobs, (1) discrete, (2) fraction 
al, and ( 3 ) multiple reaction. Figure 1 shows four knobs. 
Select one knob for each of the controls displayed on the 
next page and place its corresponding number in the blanks 
provided. 
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HEATER 


IGNITION SWITCH 


RADIO FREQUENCY 


OFF 

T 

V • ^ 

Jk 

K 


OF P 

* 


kill l r i 1 l i 

So 40 10 So 10 Ut Jit »L 0 
» 


a. 


b. 


c. 



Choice of knobs: 






8. If and only if you believe it is necessary to do so, 

redesign the dial in the space provided following recom- 
mended design practices. Put in all major and/or numbered 
markers. Minor markers need be illustrated only between 
the first two major or numbered markers. (Figure 2) 





Figure 2. Tachometer used in testing rotary equipment. The 
scale unit is -5 rpm. The operational range is 50 rpm. 
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E. The INFORMATION PROCESSING CAPABILITY Of the human operator 

is a critical aspect of design. Even though all the controls, 

displays and alarms meet their requirements, the operators 

may still experience difficulty in performing their necessary 

tasks. These difficulties may he caused by the limitations 

on human information processing capabilities. There are 

three main areas in which these capabilities may be grouped. 

One of these is PHYSIOLOGICAL PROCESSING LIMITATIONS (PPL). 
Give an example for PPL and name the other two main areas: 

1. PPL example: 

2. second area: 

3* third area: 

F. Weapon systems function in real world environments, not in 
vacuums. The operator may therefore encounter a variety of 
ENVIRONMENTAL FACTORS which may effect his/her performance. 

1. Name one resource document which provides guidelines in 

the area of environmental factors and their possible effects 
on human performance. 

2. Name five environmental factors and for each factor ex- 
plain what effect it could have on human performance: 

factor effect 

a. 

b. 

c. 

d. 

e. 
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G. There are ENVIRONMENTAL CONSIDERATIONS which need to be eval- 



uated. Three of these are (1) ATMOSPHERIC CONDITIONS, (2) NOISE, 
and (3) WORK/REST SCHEDULES. 

1. Define the term noises 



2. Give an example of an atmospheric condition which is a 
concern to and can be affected by a human factors evalu- 
ators 

« 

3- When do the greatest performance decrements occur during 
a watch- standing duty? 

a. 

b. 

4. Listed below are UNITS OF MEASURE. In the space provided 
name what each of these measures, i.e.. in. /length; lb./ 
weight. 

a . CPS/H z 

b- PSI 

c. CLO 

d. PTS/TTS 

e. RAD 

f. dS/PNDB 

g. PPM 
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H. Choose the best answer(s). Place an "X" by your choice(s). 

One or more than one answer may be correct. 

1. The overall objective(s) in taking a human factors engineer- 
ing course is(are): 

1 . to obtain the human factors engineering background 

to do your job well. 

2 . to understand a specific person's capabilities and 

limitations. 

3 . to understand the precise military procedures in- 
volved in a human factors program. 

k. to understand the importance of applying the concepts 

and priorities of human factors engineering in your 
job. 

5 * to be able to answer some questions on this test. 

2. The main reason for poor system performance is human error. 

Which (if any) of the following are contributing factors? 

1 . inadequate consideration of human performance capa- 
bilities, skill limitations and response tendencies. 

2 . little standardization of 'controls', (knobs, levers, etc.). 

3 * the fast developing pace of new technologies with 

which the human's capabilities cannot keep up. 

people with too little education and too few skills 

being brought into the armed services. 

5 * inadequate human factors input into system man- 

machine interface. 
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POSTSCRIPT 



This evaluation was originally designed to cover the entire 
course, lessons 1-40 over a period of 60 days. Murphy's Law 
was rolling full steam, however, and the evaluation had to be 
cut to 30 days and only 20 lessons. You all are the ones who 
lose out, though, because the last 20 lessons deal with quantify- 
ing human factors evaluations and practicing human factors 
concepts . 

Recently, the Government Accounting Office, the Secretary of De- 
fense and others have pointed out that the military is accepting 
weapons systems that can't fulfill their missions. They say 
that as a result we are not ready to meet our military obligations. 
Human error is pointed out as being one of the prime reasons for 
the unacceptable system performance. They also state that inad- 
equate attention to human factors design during test and evaluation 
has set up the operator to make the error. 

If we, the operational test directors, know what design factors 
can cause the human to make errors, we can help prevent them. 
Perhaps I'm a bit surrealistic, but I always thought our job was 
to deliver good and safe equipment to our military men and women 
and in doing so help save lives. 

This human factors course may have been an assignment to you -- 
maybe even one you didn't like. Whether you wanted to take the 
course or not, I sincerely hope you have learned something. If 
you have, then the course is serving its purpose and I ask you to 
please continue with the other 20 lessons. Some flyer or sailor 
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will be out there thanking you for it. Who knows, it may even be 
you. 

The course may not be perfect and it is for that reason the evalu- 
ation is being conducted. The questionnaires you will be given 
is your opportunity to tell me how to make it better. I ask for 
your continued dedication and co-operation while filling out the 
questionnaires. The feedback you provide will be invaluable. 

Once again, thank you for your help and participation. 
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AruiEX E 



CCS QUESTIONNAIRE 
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COMMAND COURSE SUPERVISOR 
QUESTIONNAIRE INSTRUCTIONS 



Directions i Based upon your observations and your log 
entries. .during the last four weeks, complete this form. 
Upon completion, submit to the Evaluation Director at the 
Naval Postgraduate School for analysis. This Evaluation 
Director may ask for your assistance and for clarification 
during the Analysis Phase of this evaluation. 



NAME COMMAND 

NUMBER OP STUDENTS DATE 



SAMPLE QUESTION . 



(3) How frequently did students’ questions/comments 
indicate : 

0-40# 41-60# 61-100# 

a. they did not like 

coffee. 

b. dissatisfaction 

with today's weather. 



If half of all students' questions/comments relating to cof- 
fee indicated they didn't like it, then place an ’X’ in the 
4l-6o# column, as shown below. 

If there are nine students’ questions/c omments relating to 
today’s weather, and three are negative, then place an 'X’ 
in the 0-40# column (3 4 - 9 = 33 * 3 $) * as shown below. 

0-40# 41-60# 61-100# 



a. they did not like 
coffee . 




b. dissatisfaction 

with today's weather. 




♦♦There are 4 pages to this questionnaire , be sure to answer all. 
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CCSQ-1 



COMMAND COURSE SUPERVISOR 
QUESTIONNAIRE 



How frequently did students' questions/comments indicate 
dissatisfaction with the: 

(a) LESSON BOOK MATERIALS: 0-40# 41-60# 61-100# 

(1) information frames 

(2) wrong answer frames 

(3) questions at end of 

information frames 

(4) answer choices at 
end of information 

frames 

(5) correct answers 

(6) course length 

(7) format 

(8) lesson length 

(9) educational level 

of course content 

(10) course content 

(11) usefulness of the 

course 

(12) amount of time 

spent on course 

(13) terminology 

(b) STUDENT SUPPLEMENT: 

(1) supplement in general 

(2) graphs 

(3) charts 

(4) symbology 

(5) legends 

(6) lesson outlines 
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CCSQ-2 



2. How frequently did students’ questions/c oraments indicate 
dissatisfaction with thei 

(c) RESOURCE DOCUMENTS t 0-40# 41-60# 61-100# 

(1) documents (in general) 

(2) 1472C 

(3) 1474B 

(d) PRETEST i 

(1) pretest (in general) 

( e ) POST TEST 

(1) post test (in general) 



3. How frequently did students' questions/comments indicate 
that initial attitudes about human factors werei 

(a) positive 

(b) negative 

(c) neutral 

4. How frequently did students' questions/comments indicate 
that final attitudes about human factors werei 

(a) postive 

(b) negative 

(c) neutral 

5. How frequently did students' questions/comments indicate i 

(a) an irritation with 

course materials. 

(b) that the course provided 

nothing new. 

(c) that human factors train- 

ing was unnecessary. 

(d) that the learning mater- 

ials were more hinder- 
ing than helpful. 

(e) that the course was a 

a waste of time. 
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CCSQ-3 



STUDENT INFORMATION 



6. How many students failed to complete 20 lessons? 

7. How many students continued beyond lesson 20? 

8. How many students completed exactly 20 lessons? 

9. How many students requested to keep the Student 

Supplement? 

10. How. many students requested to keep the entire 
course? 

ADDITIONAL COMMENTS i 
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CCSQ-4 
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STUDENT 



1UESTI0NNAIRE 
I NST RUCTIONS 



PURPOSE : The main purpose of this questionnaire is to ob- 

tain information regarding the objectives, content, design 
and presentation of the Human Factors Engineering (HFE) 
Self-Paced Course, Your answers will help to deter- 

mine what actions must be taken to improve the course and 
the quality of human factors training and education. Your 
honest opinions are, therefore, essential. 



INSTRUCTIONS : I have no need to know who you are personnally. 

No effort will be made to identify you. However, for anal- 
ysis purposes, it is necessary to have your Student Identi- 
fication number (SIF). No one besides myself will see these 
questionnaires .and no individual information will be related 
to your command. 



Before you begin, your Command Course Supervisor (CCS) will 
show you an example of an " information frame " and a " wrong 
answer frame" . This will provide clarification of terminol- 
ogy used within this questionnaire. 



Directions for answering the questions are provided below: 



SAMPLE QUESTION 

(3) What types of schools 
have you attended? Circle 
your answer (s ) . 

12 3^5 



Types of 
Schools 

1. pre-school 

2. elementary 

3 . junior high 

4. senior high 
5- 4-yr high 
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If you attended elementary school, you should circle the 
number 2 . as has been done below, since the number 2 



corresponds to elementary school. If, in addition, you also 
attended a 4-year high school, you should also circle the 
number 5 . as it corresponds to 4-year high school. 



1 





SAMPLE QUESTION ■ 

(4) In taking college courses, the college preparatory 
courses taken in your high school were 

very effective 

effective 

borderline 

ineffective 

very ineffective 



If you felt the college preparatory courses you took in 
high school did absolutely nothing to help you through 
college, place an *X* in front of "very ineffective’*, as 
shown below: 




very effective 
effective 
borderline 
ineffective 
very ineffective 



If you have any questions, please ask your CCS for assistance. 
The questionnaire must be returned to your CCS within two 
working days. 3e sure to double check that you have answered 
all questions. 
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HUMAN FACTORS ENGINEERING SELF-INSTRUCTION COURSE OBJECTIVES 



Upon completion of the Human Factors Engineering Course, the 

student will demonstrate the following capabilities and knowledge: 

1. An understanding of common terms used in human factors engineering. 

2. A familiarity with human factors references and an ability 
to use them. 

3. An awareness of potential sources of technical information on 
major human factors topics. 

4. An understanding of the goals of human factors engineering in a 
materiel development program. 

5. An ability to integrate human factors principles in a DOD 
sponsored program. 

6. An ability to determine human performance requirements in a 
systems concept. 

7. An understanding of the kinds of factors and forces which affect 
human performance and an ability to identify and measure them. 

8. An awareness of the differences between field and laboratory 
measurements . 

9. An awareness of what "experimental control" measures are necessary 
for any test involving human performance and the effects in 
their absence. 

10. An understanding of basic statistical techniques, such as 
analysis of variance. 

11. An ability to calculate human performance reliability. 

12. An ability to formulate performance measures for the dependent 
variables of time and error. 

13. An ability to analyze human performance data within the context 
of "system effectiveness" and "system reliability." 

14. An understanding of the major techniques used by human factors 
specialists during system synthesis, design, and development. 

15. A familiarity with task analyses. 

1 6 . An awareness of the relationship between human factors engineering 
and the engineering specialists of reliability, maintainability 
and safety. 

17. An ability to interpret and apply the standards and specifications 
of the human factors engineering community. 
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STUDENT 



QUESTIONNAIRE 

A. Course Content and Objectives . 

Refer to the "Course Objectives" sheet on the next page and 
answer the following by circling your answer(s). 

(1) Which of the 17 objectives listed have been supported 
(taught) by the lessons within the first half of the 
Human Factors Engineering Self-Instruction Course? 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1? 

(2) Which of the 17 objectives listed describe a skill neces- 
sary to test and evaluate a system's human factors aspects? 

1 2 3 4 5 6 7 3 9 10 11 12 13 14 15 16 17 

(3) Which of the 17 objectives listed would help you do your job? 

1 2 3 4 5 6 7 3 9 10 11 12 13 14 15 16 17 



B . Course Design 

(1) The sequence in which the various lesson topics were 
presented was 

very effective 

effective 

borderline 

ineffective 

very ineffective 



(2) The way in which each concept was developed was 



very effective 

effective 

borderline 

ineffective 

very ineffective 

(3) Direction were given to read the information presented 
in each frame, answer the question at the end of each frame 
and go to the page referenced by the selected answer. If 
the answer was incorrect, the student was referred back to 
the previous page to select another answer. If the answer 
was correct, the student proceeded with the next "information 
frame". This format was 

very effective 

effective 

borderline 

ineffective 

very ineffective 



Comment: 
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(4) How frequently was the re-enforcement or reteaching 
provided in the "wrong answer frame" sufficient to choose 
the "correct" answer? 

always frequently sometimes seldom 

(5) How frequently was the answer to the question at the 
end of each "information frame" suggested or provided in 
preceding course material? 

always 

frequently 

sometimes 

seldom 

never 

(6) How frequently was the answer to the question at the 
end of each "information frame" suggested or provided in 
course material following the question? 

always 

frequently 

sometimes 

seldom 

never 



never 
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(7) The quality of the questions at the end of each "infor- 
mation frame" was 



excellent 

good 

borderline 

bad 

very bad 

(8) The quality of the answer choices for each question at 
the end of each "information frame" was 

excellent 

good 

borderline 

bad 

very bad 

G. Course Presentation 

(9) Participation in the HPE course without the aid of an 
on-site instructor was 

very effective 

effective 

borderline 

ineffective 

very ineffective 
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(10) How frequently was assistance required of the Command 
Course Supervisor? 

always frequently sometimes seldom never 

(11) How frequently were words used which required defining 
by an outside source (dictionary, person, etc. )? 

always frequently sometimes seldom never 



D. Student Supplement 



(12) How effective/ineffective were each of the elements 
listed below in aiding your understanding of the concepts 
presented in the course material? 



Very 

effec- Effec 
tive tive 



In- 
border- effec- 
line tive 



Very 

In- 

effec- 

tive 



tables 

charts 

graphs 

illustrations 

symbology 

legends 

terminology 

lesson outlines 

arrangement 
of tables, 
graphs, etc. 
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(13) How do you feel about the student supplement as a ref- 
erence book and human factors guideline during test and eval 
uation planning stages? 

extremely useful 

of considerable use 

of use 

not very useful 

of no use 

(14-) The information given with figures and tables (charts 
and graphs) to provide clarity and meaning to each was 

extremely useful 

of considerable use 

of use 

not very useful 

of no use 

(15) The implied ideas and meanings represented by symbols 
and legends in the student supplement were 

always understood 

frequently understood 

sometimes understood 

seldom understood 

never understood 



214 



E. Resource Documents 



(16) The HFE course materials made use of the resource 
documents listed below 



very 

very border- diffi- diffi- 

easy easy line cult cult 



1472B 
1474B 
HDBK 759 



(17) Use of the resource documents listed below in planning 
human factors tests for a new system is 

very 

very border- diffi- diffi- 

easy easy line cult cult 

14723 

14743 

HDBK 759 



F. Human Factors Attitudes 

(18) How do YOU FEEL about including a human factors test 
in the evaluation of a new piece of equipment or system? 

very important 

important 

borderline 

unimportant 

very unimportant 
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G . Miscellaneous 



(19) The adventures of Lt. I. M. Eager were tracked through- 
out the HFE course. The stories about Lt. Eager were 

always effective 

frequently effective 

sometimes effective 

seldom effective 

never effective 



COMMENT : 



(20) How frequently did the factors listed below adversely 
affect your completion of the lessons in the first half of 
the course? 

always frequently sometimes seldom never 

Experience 

Education 

Time 

Job load 

Course Length 

Format 

Terminology 

Design 

TAD (temporary 

additional duty) 



(**) Please identify, in the space below, any specific 
problems you had with the following* 



LESSONS BOOKS* 



STUDENT SUPPLEMENT* 



RESOURCE DOCUMENTS: 



PRETEST* 
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POST TEST: 



(***) Would you like to keep a copy of the HFE course? 

yes no 

(***) would you like to keep a copy of the student supplement 

yes no 

(***) would you like to know your test scores? 

yes no 

(***) would you like to see the final report on the HFE 
Self-Faced Course? 

yes no 



WHAT IS YOUR Student Identification Number? 



annex g 



COURSE OBJECTIVES QUESTIONNAIRE 
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COURSE OBJECTIVES EVALUATION 



NAME 

COMMAND 

POSITION/TITLE _ 
RESPONSIBILITIES 



EXPERIENCE IN COURSE EVALUATIONS (yrs. ) (mos. ) 

TIME AT THIS COMMAND (yrs.) (mos.) 



The six characteristics listed on page one have been 
established by NAVEDTRA 106A as criteria by which in- 
structional learning objectives are to be judged. The 17 
learning objectives listed on pages two and four will, 
therefore, be evaluated according to how completely they 
meet the six criteria as explained on the next two pages. 
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Directions: Assess each learning objective according to 



each of the six criteria . Place criteria numbers (1,2,3»4,5* 
and 6) in the spaces to the right of each learning objective, 
according to how completely each objective satisfies each 
of the criteria. Each criteria should be rank ordered from 
"Not at All" to Completely". 



SAMPLE: 



A. A student will be able to 
correctly solve quadratic 
equations. 



CRITERIA FOR COURSE OBJECTIVES 

1 . Objectives must be a statement of student behavior 
(action), such as the creation of a product or some 
other overt act, which can be accepted as evidence that 
the intended outcome has occurred. 

2. The behavior must describe specifically all outcomes 
that will demonstrate that learning has occurred. 

3 . The student behavior called for must be capable of 
observation and evaluation within the learning and 
testing environments. 

4. The objective must be stated in learner rather than 
"teacher" terms, i.e., actions which the student will 
perform rather than what the instructional materials 
will "say or do". 

5- There must be a standard against which the student 

behavior will be measured. It must be fully specified. 

6 . The statement of the conditions under which the student 
behavior will occur must be fully specified. 
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If you feel that objective "A” fulfills criteria 1,2,3, 
and 4 completely, then place the numbers 1,2.3. and 4 in the 
blank under "Completely " as shown below. If you feel that 
objective "A" does not meet criteria 5 at all, place a "5 " 
under "Not at All 1 *. If it meets criteria 6 to some extent, 
but needs quite a bit of attention, then place a 11 6" under 
"To Some Degree ' 1 . as show below. 



A. A student will be able to 
correctly solve quadratic 
equations. 




If you have any questions or comments regarding this 
evaluation, please feel free to contact me or write them 
in the space provided at the end of this evaluation. 



Lt. Martha M. Fleming 
Naval Postgraduate School 
SMC $ 1340 

Monterey, California 93940 

Autovon: 37S-2536 

Commercial? 408-646-2536 
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LEARNING OBJECTIVES 



1. An understanding of common terms 
used in human factors engineering. 

2. A familiarity with human factors 
references and an ability to use 
them. 

3 . An awareness of potential sources 
of technical information on human 
factors topics. 

An understanding of the goals of 
human factors engineering in a 
material development program. 

5 . An ability to integrate human 
factors principles in a Department 
of Defense sponsored program. 

6 . An ability to determine human per- 
formance requirements in a systems 
concept . 

7. An understanding of the kinds of 
factors and forces which affect 
human performance and an ability 
to identify and measure them. 

8 . An awareness of the differences 
between field and laboratory 
measurements . 

9. An awareness of what "experimental 
control" measures are necessary for 
any test involving human perform- 
ance and the effects in their ab- 
sence. 

10. An understanding of basic statis- 
tical techniques, such as analysis 
of variance. 

11. An ability to calculate human per- 
formance reliability. 

12. An ability to formulate perform- 
ance measures for the dependent 
variables of time and error. 
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CRITERIA FOR COURSE 03JECTIV5S 



1. Objectives must be a statement of student behavior 
(action), such as the creation of a product or some 
other overt act, which can be accepted as evidence 
that the intended outcome has occurred. 

2 . The behavior must describe specifically all outcomes 
that will demonstrate that learning has occurred. 

3 . The student behavior called for must be capable of 
observation and evaluation within the learning and 
testing environments. 

4. The objective must be stated in learner rather than 
"teacher" terms, i.e., actions which the student will 
perform rather than what the instructional materials 
will "say or do". 

5 . There must be a standard against which the student 
behavior will be measured. It must be fully specified. 

6 . The statement of the conditions under which the student 
behavior will occur must be fully specified. 



NA7EDTRA 106A 
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LEARNING OBJECTIVES 



13 . An ability to analyze human per- 
formance data within the context 
of "system effectiveness" and 
"system reliability". 

14. An understanding of the major 
techniques used by human factors 
specialists during system synthesis, 
design, and development. 

15 . A familiarity with task analyses. 

16. An awareness of the relationship 
between human factors engineering 
and the engineering specialists of 
reliability, maintainability and 
safety. 

17 . An ability to interpret and apply 
the standards and specifications 
of the human factors engineering 
community. 



A. Are these the criteria by which you judge learning objectives? 
(Circle one): YES NO 



If the above answer was "NO", please provide a copy of the 
criteria you do use. ) 



3. Do you approve of these 6 characteristics being used as 
criteria for judging learning objectives? (Circle one): 

YES NO 



C . Why? : 
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APPENDIX F 



DEMOGRAPHIC AND PRE-POST TEST DATA 
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Demographic and Pre-Post Test Score Data for the 
Control Group at Each of Four Test Sites 

(n=26) 



p 


co 


CO 


P 


1) 


CP 


H 


CD <P 


P 


P O 


CO 


O 


o 


CJ 


CP CO ' — ' 




/ — \ 




• 




CO 




p 


p 


CP 


CO 




0) 


CD MH 


p 


P O 



<D 



G CO ' 



<d 

u 

r* 

s 

•H 
W M 
0 

or G 
X 

H W 

00 

e 



W 03 
W P. 
G E- 



o 

CCS 

o o 

• H US’ 
CJ «P 
S 03 CO 
•H +J P 
h W X 



C 

o 

•H 

P 

03 

U 

3 

^3 

W 



0) 

"T 3 

03 

P 

00 

03 

G 



CP 

H— I 

co 



, 5 
, 5 
, 5 


,0 
,5 
, 5 


pH LO 


LO o 


LO O LO 


o 


LO 


LO 


, 5 
.0 
, 0 
, 5 


LO 


G 


LO 


LO LO LO 


o\ cr> 


0% tO vO 


l o lo 


\o 


h' rj* 


oo 


LO 


LO 


eg g nr o 


OO 


LO 


O rH 


nT LO H 


N in 


to 


LO 


tO H ^ 


nr 




to 


LO LO LO tO 


to 


r*- 


nr nT lo 


o o o 


O O o 


o o 


o o 


,0 
, 0 
, 0 


LO 


G 


o 


G G G G 


o 


O 


G 


LO G G 


nr o vo 


VO rH o 


oo 


OO rH 


Ol N h* 


r- 


o 


r>- 


O G O tO 


eg 


to 


r- 


lo eg to 


O OO vO 


lo r-- 


oo oo 


oo 


om^r 

rH 


G 






lo oo r- lo 


G 


oo 


r^- 


O OO c 


E 

E 

E 


C C UJ 


03 w 


E 

E 


WWW 


w 


w w 


E 

E 

E 

E 


w 


W 


W 


WWW 


US’ US’ US* 


US’ us* US’ 


US’ US’ US’ US’ 


US’ US’ US’ 


US’ 


US’ US’ 


US’ US’ US’ US’ 


US’ 


US’ 


US’ US’ US’ US’ 


E- H E- 


hhh 


H H 


H E- 


T 

T 

T 


H 


H 


E- 


T 

T 

T 

T 


H 




H 


T 

T 

T 


CO CO 00 


HO CO CO 


CO CO 


CO CO 


CO CO CO 


CO 


CO 


CO 


CO CO CO CO 


CO 


W 






PC pc pc 


PC PC c^ 


pc PC 


PC PC 


PC PC PC 


PC 


PC 


PC 


PC PC PC PC 


PC 


G 






cj cj cj 


a cj cj 


cj cj 


CJ CJ 


uuu 


CJ 


cj 


CJ 


uuuu 


CJ 




H 


8—E—8— 






















PC 




^ *-3 *-> 


Q Q Q 


g g a 


a o 


G Q 


a a g 


Q 


G 


G 


Q Q Q G 


G 




O 


o o o 


H H E- 


hhh 


E- H 


H H 


E- E- E- 


E-* 


E- 


H 


E- E- E- H 


E- 








o o o 


o o o 


O O 


O O 


O O C 


O 


O 


O 


O O O O 


O 


eg 






O nT r-H 


o Ol o 


to 


G eg 


6 

0 

0 


o 


o 


o 


o o oo 


rH 


eg 


G 


o rr so 


pH 


r— t 








rH 














rH 


•3 

-I 

-I 


tO eg 


rH eg 


eg rg 


1- 

2- 

2- 


CvJ 


o 


G 


•i 

■i 

•i 

■z 


rH 




to 
















eg 


rH 




















rH 




















• • 


• 


• 


• o 




• 




• • 


• 


• 




• 




00 00 


00 


oo 


00 CJ 


• 


oo 


oo oo • • 


00 


00 




• 00 




CD CD 


CD 


CD 


CD CO 


rH 


0) 




CJ (D rH rH 


CJ 


CJ 




rH CJ 


CO CO CO 


CCw 


q co 


CO G 


CO G 


rH 


G 


CO 


Q Q rH rH 


Q 


G 


CO 


rH G CO 


ocu 


• • cj 


• CJ 


CJ • 


cj • x: 


o 


• 


CJ 


• • o o 


• 


• 


CJ 


O • CJ 


CP CP CP 


pH rH CP 


rH Pm 


CP rH 


G »h OOCJ 


r-H 


pp 


rl rH CJ CJ 


rH 


rH 


G 


CJ rH G 




o o 


o 


o 


o -h 




o 




O O 


o 


o 




O 




cj cj 


CJ 


CJ 


CJ G 


(VJ 


CJ 




UUfMN 


CJ 


CJ 




to CJ 














LO 


nr 


w w 










•5 

-5 

■4 


-4 

-4 

•4 


nT nT 


nr nr 

i i 

o o 


■3 

•3 

•8 


G 


rH 


rH 


to to to h- 


LO 


Cl 


to 


•3 

•5 

■2 


• 1 1 

o o o 


• • • 
o o o 


O O 


0- 

0- 

E- 


1 

w 


1 

CO 


CO 


0- 

0- 

0- 

E- 


w 


1 

w 


1 

G 


W CO G 














CJ 


CJ 










CJ 


oo cr> 


OHM 


to nr 


LO VO 


oo o 


o 


rH 


OvJ 


G oo cr> 


G 


eg 


vO 


c — oo to 


pH p-h iH 


CV1 CM <VJ 


eg eg 


CvJ (VJ 


eg eg rvj 


to 


to 


to 


G O O O 


f-T 


G 


O 


G G G 


rl 
















eg 




to 


nr 




O 
















G 




G 


o 





ii ii 

Ixc 

co 



227 



70.19 X = 49.70 
14.85 S . D . = 16.48 



inographic and Pre-Post Test Score Data for the 
Course Group at Each of Four Test Sites 
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Completed at least 20 lessons. X = 64.73 X = 93.39 

S . D. = 18.70 S . D. = 25.14 



APPENDIX G 



POST TEST RAW SCORES 
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Post Test Raw Scores for the Control Group 
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Post Test Haw Scores for the Course Group 
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APPENDIX H 



STUDENT ATTITUDE SURVEY 
(QUESTIONNAIRE) 

DATA 
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Table 28 



Record of Student Attitude (Questionnaire) Responses 

A. COURSE CONTENT AND OBJECTIVES 

1. Objectives taught in the first 20 lessons: 



Obj. #: 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


Responses : 


20 


19 


11 


16 


10 


11 


14 


7 


7 


2 


Obj . # : 


11 


12 


13 


14 


15 


16 


17 








Responses : 


2 


3 


2 


10 


10 


6 


8 








Obj ectives 


necessary 


■ to 


do 


HFE 


Test 


and 


Evaluation : 


Obj . # : 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


Responses : 


8 


8 


7 


7 


9 


9 


10 


5 


9 


7 


o 
c r 

V— 1. 
• 


11 


12 


13 


14 


15 


16 


17 








Responses : 


11 


10 


12 


S 


9 


8 


6 








Obj ectives 


which 


. would help 


you 


do 


your 


job : 






Obj. #: 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


Responses : 


10 


12 


11 


9 


10 


9 


12 


6 


9 


8 



Obj . # : 


11 


12 13 14 


15 16 17 


Responses : 


8 


7 13 9 


9 13 6 
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Table 28 - Continued 



B. COURSE DESIGN 



Student Responses 





Most 

Positive 
5 4 


3 


2 


Most 

Negative 

1 






/ 


/ 


/ 


/ 


/ 


Total 


1. Sequence of les- 


son topics 


2 


14 


4 


1 


3 


24 


2. Concept develop- 


ment 


1 


13 


5 


2 


3 


24 


3. Format 


1 


11 


2 


5 


5 


24 


4. Remediation 


1 


6 


9 


3 


4 


23 


5. Info provided 


before questions 


1 


11 


9 


2 


1 


24 


6. Info provided 


after questions 


2 


5 


14 


1 


2 


24 


7. Question quality 

8. Answer choice 


1 


11 


10 


1 


1 


24 


quality 

COURSE PRESENTATION 


1 


10 


10 


1 


2 


24 


9. Course w/o on-site 


instructor 


1 


8 


7 


4 


4 


24 


.0. Freq. use of CCS 


15 


6 


3 


- 


- 


24 


.1. Terminology- -defs . 


provided 


6 


9 


7 


2 


- 


24 
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Table 28 - Continued 



D. STUDENT SUPPLEMENT 



Student Responses 

Mo st Most 

Positive Negative 

5 4 3 2 1 



/ / / / / Total 



Effectiveness of course 


elements 








Tables 


2 


15 


6 


- 


- 


23 


Charts 


2 


14 


7 


- 


- 


23 


Graphs 


2 


13 


8 


- 


- 


23 


Illustrations 


2 


15 


6 


- 


- 


23 


Symbology 


- 


12 


8 


2 


1 


23 


Legends 


- 


14 


7 


1 


- 


22 


Terminology 


- 


15 


6 


- 


2 


23 


Lesson outlines 


3 


10 


8 


1 


1 


23 


Arrangement of 
tables § graphs 


- 


12 


7 


3 


1 


23 


Usefulness of Stu- 
dent Supplement as 
a reference 




6 


10 


7 


1 


24 



Student Supplement 
(usefulness § self- 
explanatory) --fig- 
ures § tables 


8 


13 


3 


- 


24 


Clarity of symbols 


and legends 4 


11 


7 


1 


1 


24 
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Table 28 - Continued 



E. RESOURCE DOCUMENTS 



Student Responses 



Most 

Positive 



Most 

Negative 



5 4 3 2 1 

/ / / / / Total 



16. Course made reference documents easier/harder to use: 



1472C 
1474B 
HDBK 759 



2 8 7 2 
2 7 5 2 
2 11 5 2 



19 
16 

20 



17. Use of ref documents in planning HFE T 8 E made easier/ 
harder : 



1472C 


- 


10 


1474B 


- 


8 


HDBK 759 


- 


11 



6 3 - 19 

6 3 17 

5 3 19 



F. HUMAN FACTORS ATTITUDES 

18. Is HFE T 8 E necessary: 

12 10 1 1 24 

G. MISCELLANEOUS 

19. Story Line Effectiveness: 

1 3 10 4 6 24 
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Table 28 



Continued 



G. MISCELLANEOUS (cont'd) 



(Never) 

Most 

Postive 



Student Responses 

(Always) 

Most 

Negative 



5 4 3 2 1 

/ / / / / Total 



20. Factors Adversely affecting completion of lessons: 



Experience 


10 


5 


5 


3 


- 


23 


Education 


10 


4 


5 


3 


1 


23 


Time Available 


1 


1 


9 


9 


3 


23 


Job Load 


1 




7 

/ 


11 


3 


22 


Course Length 


6 


6 


5 


4 


2 


23 


Format 


3 


6 


5 


1 


9 


24 


Terminology 


8 


2 


9 


5 


- 


24 


Design 


5 


4 


3 


4 


5 


22 


Temporary Addi- 
tional Dury 


6 


3 


8 


5 


2 


24 



COURSE DESIRABILITY 




















Yes 


No 


Total 


Students 


want 


to 


keep 


course materials. 


10 


14 


24 


Students 


want 


to 


keep 


Supplement only. 


13 


11 


24 
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APPENDIX I 



QUESTIONNAIRE DATA 
FROM CURRICULUM DEVELOPMENT EXPERTS 
ON TERMINAL LEARNING OBJECTIVES 
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Not at all S = To some degree M = Mostly C = Completely 






7 - An understanding of kinds of 8 . An awareness of the differ- 9 • An awareness of what "exper- 

factors and forces which affect hu- ences between field and labora- mental control 1 * measures are ne- 

man performance and an ability to tory measurement. cessary for any test involving 

identify and measure them. human performance and the effects 

in their absence 



1 
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Not at all S = To some degree M = Mostly C = Completely 
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Not at all S = To some degree M = Mostly C = Completely 






APPENDIX J 



SAMPLE RECOMMENDATIONS 
ON COURSE LAYOUT 



242 



Figure 1 

Module Cover Page Format 



MODULE 1.0 



HUMAN CAPABILITIES AND LIMITATIONS 

FOR 

HUMAN FACTORS ENGINEERING 



Course Identification Number 



Prepared by 

Pacific Missile Test Center, Pt. Mugu, California 

25 December 1983 
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Figure 2 

Module Overview Format 



MODULE OVERVIEW 
MODULE 1.0 



Human Factors Engineering 
HUMAN CAPABILITIES AND LIMITATIONS 



In this module 



List of Lesson Topics Included in Module: 

Lesson Topic 1 - Welcome to Human Factors Engineering 
Average Time: 

Lesson Topic 2 - Why Human Factors? 

Average Time: 

Lesson Topic 3 - Tragic Mistakes ... and Positive 

Consequences . 

Average Time: 



Lesson Topic 20 - Review 

Average Time: 

Total Module Average Time: 

(Insert any general instructions to students concerning the 
use of this Module Booklet and any of its supporting mater- 
ials . ) 



244 




Figure 3 

Lesson Topic Cover Page Format 



HUMAN FACTORS ENGINEERING 

MODULE NUMBER 1.0 
LESSON TOPIC 1.1 

WELCOME TO HUMAN FACTORS ENGINEERING 



25 December 1983 
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Figure 4 

Lesson Topic Overview Format 



MODULE 1.0 
LESSON TOPIC 1.1 

LESSON TOPIC OVERVIEW 
LESSON TOPIC 1.1 

WELCOME TO HUMAN FACTORS ENGINEERING 
In this Lesson Topic . . . 

The Learning OBJECTIVES of this Lesson Topic are as 
follows : 

1 . 

2 . 

3. 

4. 

(Statement to the student that he or she should review the 
"LIST OF STUDY RESOURCES” and read the Lesson Topic LEARN- 
ING OBJECTIVES before beginning the Lesson Topic.) 



246 




Figure 5 

List of Study Resources Format 



MODULE 1.0 
LESSON TOPIC 1.1 



LIST OF STUDY RESOURCES 
WELCOME TO HUMAN FACTORS ENGINEERING 



To learn the material in this LESSON TOPIC, you have the 
option of choosing, according to your experience and 
preferences, any or all of the following study resources. 



Written Lesson Topic presentations in the Module Booklet: 



1. Lesson Topic Summary 

2. Programmed Instruction Form of Lesson Topic 

3. Student Supplement containing supporting 
information, charts, graphs, etc. 

4. Lesson Topic Progress Check 

5. Narrative Form of Lesson Topic 



Additional Materials 



1. Student Response Sheets 

a. Diagrams, charts, graphs 

b. Programmed Instruction Response Sheets 

c. Answer Sheet for use with all tests 

d. Notetaking Sheets 

Enrichment Materials 

1. Additonal Resource Documents from the Student 
Packet 

2. Additonal References from the Naval Supply 
System 

You may use any or all resources listed above, including 
the Learning Course Sponsor, but all materials listed are 
not necessarily required to achieve Lesson Topic Objectives 
The Progress Check may be taken at any time. 
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Figure 6 

Lesson Topic Summary Format 



MODULE 1.0 
LESSON TOPIC 1.0 

LESSON TOPIC SUMMARY 



Welcome to Human Factors Engineering 



(A condensation of the narrative form of the lesson topic 
is placed here) . 



Statement to the student: 

"At this point, you may take the Lesson Topic Progress 
Check. If you answer all self-test items correctly, pro- 
ceed to the next Lesson Topic. If you incorrectly answer 
only a few of the Progress Check Questions, the Correct 
Answer page will refer you to the appropriate pages, para- 
graphs, or frames so that you can restudy the parts of this 
Lesson Topic you are having difficulty with. If you feel 
that you have failed to understand all, or most, of the 
Lesson Topic, select and use another medium of instruction: 
Narrative or consultation with the Learning Course Sponsor, 
until you can answer all self-test items on the Progress 
Check correctly." 
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